qualitative research systematic review

Jump to navigation

Cochrane Training

Cochrane-campbell handbook for qualitative evidence synthesis.

Version 1.0, 2023

The Cochrane-Campbell Handbook for Qualitative Evidence Synthesis  is the official guide that describes in detail the process of preparing and maintaining systematic reviews of qualitative evidence for Cochrane and Campbell reviews. The Handbook has been produced by the Cochrane Qualitative and Implementation Methods Group  and members of the Campbell Qualitative Evidence Synthesis working group . It is a step-by-step guide for those conducting systematic reviews of qualitative evidence and a reference for more experienced authors.

The Handbook is applicable to all systematic reviews of qualitative evidence, though it is specifically relevant to Cochrane and Campbell Reviews. Part 1 covers the core methods used in Cochrane systematic reviews of qualitative evidence. Part 2 introduces a collection of chapters detailing other relevant methods to consider for the systematic review of qualitative evidence. These methods have not yet been fully tested as core methods for Cochrane Reviews. The chapters in Part 2 cover methods that are either a) established within the wider systematic review community but have not yet been fully utilised in Cochrane or Campbell Reviews or b) relatively novel but with a significant body of published methodological work behind them. Part 3 covers how to report a systematic review of qualitative evidence and provides guidance on how to peer review a qualitative evidence synthesis. 

Chapters are available below for personal use via a Cochrane Account (don't have an account? Set one up for free here ). 

Part 1: Core methods

• Starting a qualitative evidence synthesis 
• Defining the review scope and formulating review questions
• Selecting and using theory
• Developing and using logic models
• Searching for and identifying studies
• Selecting studies and sampling
• Assessing study methodological strengths and limitations
• Selecting a method of synthesis and data extraction
• Conducting a framework synthesis
• Conducting a thematic synthesis
• Conducting a meta-ethnography
• Using visual methods to support synthesis
• Assessing confidence in the evidence using the GRADE-CERQual approach
• Integrating qualitative and quantitative evidence

Part 2: Other relevant methods

• Conducting time-sensitive reviews
• Conducting a realist synthesis
• Reviewing diverse types of implementation evidence
• Conducting a qualitative comparative analysis  
• Introducing meta-narrative reviews, critical interpretive synthesis, narrative synthesis and meta-aggregation

Part 3: Reporting and peer review

• Reporting a protocol and a review
• Peer reviewing a protocol or a review

Important notes

• A book version (first edition) of this Handbook , to be published by Wiley in 2024, will comprise chapters 1–21. The online version will include chapters 1–21 with supplements.
• Chapters are freely available for personal use only via a Cochrane Account (don't have an account? Set one up here ). These chapters are not for general distribution, and all content remains the copyright of Cochrane.
• These drafts will undergo further work. Further drafts may be posted here before finalization.
• If you would like feedback on any draft chapters, please use this Form . Your comments will be considered in finalising the drafts for the printed copy of the handbook. 
• If you use the chapters in your work, please cite the material as set out at the beginning of each chapter, including the exact date of the version.

Editorial team

Jane Noyes (Senior Editor)  Angela Harden (Senior Editor)  Heather Ames (Associate Scientific Editor)  Andrew Booth (Associate Scientific Editor)  Kate Flemming (Associate Scientific Editor)  Emma France (Associate Scientific Editor)  Ruth Garside (Associate Scientific Editor)  Catherine Houghton (Associate Scientific Editor)  Tomas Pantoja (Associate Scientific Editor)  Katy Sutcliffe (Associate Scientific Editor)  James Thomas (Associate Scientific Editor) 

Back to top

• Open access
• Published: 04 May 2016

Searching for qualitative research for inclusion in systematic reviews: a structured methodological review

• Andrew Booth 1  

Systematic Reviews volume  5 , Article number:  74 ( 2016 ) Cite this article

60k Accesses

135 Citations

97 Altmetric

Metrics details

Qualitative systematic reviews or qualitative evidence syntheses (QES) are increasingly recognised as a way to enhance the value of systematic reviews (SRs) of clinical trials. They can explain the mechanisms by which interventions, evaluated within trials, might achieve their effect. They can investigate differences in effects between different population groups. They can identify which outcomes are most important to patients, carers, health professionals and other stakeholders. QES can explore the impact of acceptance, feasibility, meaningfulness and implementation-related factors within a real world setting and thus contribute to the design and further refinement of future interventions. To produce valid, reliable and meaningful QES requires systematic identification of relevant qualitative evidence. Although the methodologies of QES, including methods for information retrieval, are well-documented, little empirical evidence exists to inform their conduct and reporting.

This structured methodological overview examines papers on searching for qualitative research identified from the Cochrane Qualitative and Implementation Methods Group Methodology Register and from citation searches of 15 key papers.

A single reviewer reviewed 1299 references. Papers reporting methodological guidance, use of innovative methodologies or empirical studies of retrieval methods were categorised under eight topical headings: overviews and methodological guidance, sampling, sources, structured questions, search procedures, search strategies and filters, supplementary strategies and standards.

Conclusions

This structured overview presents a contemporaneous view of information retrieval for qualitative research and identifies a future research agenda. This review concludes that poor empirical evidence underpins current information practice in information retrieval of qualitative research. A trend towards improved transparency of search methods and further evaluation of key search procedures offers the prospect of rapid development of search methods.

Peer Review reports

The contribution of qualitative evidence to health care decision-making is increasingly acknowledged. Qualitative evidence syntheses (QES) now occupy an important role within the activities of international collaborations, such as the Cochrane Collaboration [ 1 ], as part of the guidance production processes of national organisations such as the UK National Institute for Health and Clinical Excellence (NICE) and the US Agency for Healthcare Research and Quality (AHRQ) and as a genuine academic endeavour funded by private, public and charitable funding bodies. Increasingly, QES are viewed as a putative mechanism by which the systematic review “catechism” can be advanced from “what works” to “what happens” [ 2 ]. Milestones for the development of QES methodology are well-documented [ 3 ]. They include the publication of the first methodology for qualitative synthesis (meta-ethnography) in 1988 [ 4 ], the formal recognition of the Cochrane Collaboration’s Qualitative Methods Group in 2006 and publication of the first Cochrane QES in 2013 [ 5 ].

Data compiled for the annual Evidence Synthesis of Qualitative Research in Europe (ESQUIRE) workshop in 2015 suggests that between 40 and 70 qualitative syntheses are published each month across a wide range of disciplines with 2–5 methodological references on qualitative synthesis appearing within the same period.

The most cited QES methodological guidance is Chapter 20 of the Cochrane Handbook, authored by co-convenors of the Cochrane Qualitative Methods Group [ 6 ]. This was the first document to recognise the potentially important role of qualitative research within the Collaboration. Space constraints limited the searching section to three paragraphs which covered the usefulness of filters, the importance of supplementary searching strategies, an early attempt to highlight the importance of sampling decisions and a cursory sentence on reporting standards. Following receipt of a methodology grant and subsequent methodological summit in Adelaide, the Cochrane Qualitative Methods Group produced supplementary guidance hosted on the Group’s Website. Chapter 3 of this supplementary guidance covered searching for studies [ 7 ] mirroring the trial-focused chapter on searching from the Cochrane Handbook. Then, Cochrane policies confined qualitative research to a supporting role within collaboration activities [ 3 ] which resulted in potentially useful guidance on supplementary approaches to searching being relegated to an Appendix .

At about this time, the Centre for Reviews and Dissemination was revising its guidance on conducting systematic reviews. For the first time, a chapter on qualitative systematic reviews was included in this seminal guidance. Chapter 6 entitled “Incorporating qualitative evidence in or alongside effectiveness reviews” consisted of 20 pages including just over two pages related to identification of the evidence [ 8 ]. Topics covered included a characterisation of search procedures, single paragraphs on sampling approaches and supplementary strategies, respectively, a lengthy discussion of search strategies and filters and a single sentence on reporting standards.

Despite considerable advances in QES methodology, many gaps remain to be addressed. While this is true for all stages of the review process, the place of searching at the beginning of the process renders it a particular priority. Our knowledge of searching for qualitative research is founded primarily on custom and practice. Very few empirical studies exist to inform information retrieval practice. Consequently, we have an imperfect knowledge of the most effective retrieval terms, partial understanding of the respective yield of different sources and, in particular, an incomplete insight of the appropriateness of different sampling methods as they relate to different types of QES.

This methodological review was compiled to support the work of the author and other co-convenors of the Cochrane Qualitative and Implementation Methods Group in writing updated guidance on literature searching for qualitative evidence. In conjunction with a pending major revision of the Group’s chapter in the Cochrane Handbook, the co-convenors of the Group have developed a publishing plan for supplementary guidance, including a chapter on searching. This methodological overview does not duplicate the forthcoming guidance. It documents the evidence base that will inform the guidance, much as a systematic review might inform subsequent clinical guidelines. The aim is to produce a summary of the evidence base for searching for QES that is not constrained by current interpretations of the role of QES within the Cochrane Collaboration. Such a methodological summary may conceivably inform handbooks and other guidance as produced by health technology assessment (HTA) agencies, guideline producers and other review organisations.

Specifically, the author sought to address three methodological questions:

What is the current state of knowledge in relation to this aspect of searching practice?

How robust is the evidence base for this aspect of searching practice?

What are the main gaps and future research priorities for this aspect of searching practice?

Systematic approaches to searching may be typified by seven characteristics (“the 7S structure”). As highlighted in a recent paper, systematic retrieval requires a transparent method for producing a structured review question, the availability of search strategies (or filters) to assist in sifting relevant studies from those likely to be irrelevant, and replicable and evidence-based search procedures that can be enhanced and adapted to each particular review [ 9 ]. An additional challenge is associated with choosing which sources to search and learning the idiosyncrasies of each source [ 10 ]. These requirements extend to three further characteristics—standards for reporting search strategies, an informed strategy for sampling studies and judicious use of supplementary search methods. These seven characteristics were used to structure findings from the review, following a summary of existing overviews and guidance for information retrieval in QES.

Typically, a methodological review requires the conduct of sensitive searches across multiple databases. However, for the last 7 years, the author has maintained the study register of the Cochrane Qualitative and Implementation Methods Group. This is populated on a monthly basis by sensitive keyword searches of PubMed MEDLINE ( Appendix ) and the ISI Web of Science and by citation searches for key methodological books and journals on ISI Web of Science and Google Scholar. Reference lists from new articles and book chapters are checked regularly for further additions to the register. The register is the single most comprehensive database source of (i) methodological references on all aspects of QES and (ii) published examples of QES. A sensitive search was therefore conducted of the study register on Reference Manager 12 using such terms as “search*,” “retriev*” and “database*.” Six hundred fifty-four references were retrieved, and these were reviewed for relevance by the author. A high proportion of references were anticipated to be “false hits” as they reported the search methods used for specific QES. However, this sensitive search strategy increased confidence that all relevant methodological sources would be retrieved.

Previous guidance on searching for qualitative evidence was used to compile a list of 15 key citation pearls on various aspects of retrieval. A citation pearl is an authoritative article, typically identified by experts, of particular relevance to the topic of inquiry that can be used to search for relevant and authoritative materials sharing common characteristics with the original pearl [ 11 ]. Searches on Google Scholar using each title in quotation marks were used to identify all references citing these pearls (i.e. as indicated with “Cited By”). Including duplicates, 1063 references were identified from these 15 citation pearls (Table  1 ). Therefore, 1717 references were identified through the two search approaches. Once duplicates were removed, 1299 records were available.

Relevant references were coded under one or more of eight headings used to structure this review, namely Overviews and Methodological Guidance, Sampling, Sources, Structured Questions, Search Procedures, Search Strategies and Filters, Supplementary Strategies and Standards. Searching, sifting and coding were conducted in July/August 2015 [ 10 , 12 – 25 ].

The author examined the full-texts of all items identified for inclusion from the searches, most having been previously assembled to support the methodological work of the Cochrane Qualitative and Implementation Methods Group. Two types of evidence were used in compiling this structured methodological review. Empirical studies were examined to assess their practical implications for those conducting QES. Methodological commentaries, overviews and guidance handbooks were inspected in order to construct a snapshot of current practice.

Data to answer the three review questions was extracted into a single spreadsheet using Google Forms. The 7S framework was used as a structure for data extraction. Narrative text extracts from each article were cut and pasted into the data extraction framework. In addition, papers were categorised for study design and empirical studies examined for the quality of their design. A narrative commentary was produced to summarise both the included references and findings from the extracted data.

A total of 113 items were identified for inclusion in the methodological review [ 10 , 12 – 120 ]. Of these, 46 were characterised as overviews of QES methodology, 13 represented formal guidance on conduct or reporting with a further 3 being narrative reviews specifically of the QES search process. One paper was a short general summary and could thus not be classified as an overview in the literal sense. Seven papers used a hybrid design that combined an overview with one ( n  = 4) or multiple ( n  = 2) case studies and, in one case, with both a survey and a case study. The remaining 43 papers employed a formal study design and are described more fully below.

With regard to the seven components of the 7S framework, the papers were distributed as follows: Sampling ( n  = 47), Sources ( n  = 22), Structured Questions ( n  = 17), Search procedures ( n  = 6), Search Strategies and Filters ( n  = 16), Supplementary Strategies ( n  = 24), and Standards ( n  = 17). The aggregate number of papers exceeded 131 items indicating that some papers, particularly overviews, contributed to more than one of the 7S components (Fig.  1 —flow chart). Although conclusions based simply on “vote-counting” should be resisted, it is noteworthy that issues of sampling are frequently discussed, in marked contrast to a default of comprehensive sampling typically used when conducting quantitative systematic reviews. In comparison with quantitative systematic reviews, there was also greater emphasis on the use of a variety of sources and of supplementary search methods.

PRISMA 2009 flow diagram

With regard to the quality of the included papers, it was not possible to perform quality assessment for the body of included papers because (i) many papers were narrative offerings ( n  = 68) and (ii) even where a formal study design ( n  = 43) was present, the heterogeneity of these designs and the lack of a common appraisal instrument made comparability across studies prohibitively challenging. Observations on the robustness of the evidence base for each aspect of searching practice were therefore constrained to types of supporting designs. Specific observations are included within each of the following component sections. However, overall, the studies can be characterised as follows: case study (25); comparative study (7); literature surveys (5); multiple case studies (1); surveys (1); and validation study (4). Unsurprisingly, case studies of individual QES were the most common investigative design with a single paper reporting multiple case studies within the same paper. Five literature surveys identified a set of eligible studies and then examined reporting of methods within the study set. One study surveyed those who conduct literature searches. In terms of more robust designs, seven studies used a comparative design and four studies investigating the use of search filters attempted validation against a gold standard.

Many authors document the perceived complexity of searching for qualitative research, either in comparison with quantitative research (particularly randomised controlled trials) [ 8 , 14 , 36 , 39 ] or in its own right [ 10 , 25 ]. As a consequence, searching for, and identifying, appropriate qualitative research is characterised as “frustrating and difficult” [ 22 ]. Table  2 summarises the most frequently documented challenges.

While some challenges are being addressed, for example with the addition of the Medical Subject Heading (MeSH) term to MEDLINE in “Qualitative Research” in 2003 [ 34 ] and with the trend towards more informative titles and abstracts, progress appears slower than in the more researched domain of quantitative research. Established methods for identifying quantitative research do not necessarily translate into effective strategies for qualitative research [ 34 ]. Dixon-Woods et al. [ 30 ] reported that 23 % of records screened for a qualitative review of support for breastfeeding did not include an abstract. As a consequence, a higher proportion of full-text articles may need to be screened to make decisions about inclusion [ 45 ].

Overviews, summaries and guidance

For inclusion in this section, a publication (i.e. peer-reviewed book chapter, journal article or report) was required to either (i) provide an overview of the entire literature searching process, or substantive components of this process, in the specific context of qualitative research or (ii) attempt a methodological overview or analysis of one or more methods of qualitative synthesis including a consideration of literature searching methods. Forty-six items were overviews of QES methodology [ 14 , 15 , 18 , 26 – 28 , 30 – 32 , 35 , 36 , 38 , 43 , 47 – 51 , 53 – 55 , 57 – 59 , 61 , 63 , 64 , 66 , 67 , 69 , 70 , 75 – 79 , 83 , 85 , 86 , 89 , 91 , 98 , 99 , 102 , 103 , 113 ], and further seven overviews were combined with one [ 10 , 29 , 41 , 52 ] or multiple case studies [ 37 , 40 ] and, in one instance, with both a survey and a case study [ 46 ]. Thirteen papers represented guidance on conduct or reporting [ 6 – 8 , 33 , 34 ] [ 80 , 88 , 97 , 100 , 117 – 120 ], three were narrative reviews of the QES search process only [ 106 , 111 , 112 ] and there was one general summary [ 71 ].

These overview texts reflect a range of approaches to literature searching. Some simply translate the comprehensive model for quantitative studies to a qualitative context [ 47 ]. However, increasingly, overviews acknowledge differences between aggregative and configurative reviews [ 48 ] and the respective merits of comprehensive and purposive sampling [ 49 ].

For inclusion in this section, a publication was required to either (i) provide an overview of sampling in the context of QES or (ii) include a substantive discussion of the topic of sampling within a published synthesis, or (iii) mention sampling within works identified for the “ Overviews, summaries and guidance ” section. Forty-seven items were thus included in this section [ 4 , 6 , 8 , 10 , 12 , 30 , 32 , 34 , 35 , 37 , 38 , 40 , 42 , 45 – 47 , 49 – 66 , 68 – 76 , 78 – 80 , 82 ].

While there is general agreement on the need for search strategies to be systematic and explicit, recent debate focuses on whether QES share the need for comprehensive, exhaustive searches [ 69 ]. Some argue that a more purposive sampling approach, aiming to provide a holistic interpretation of a phenomenon, where the extent of searching is driven by the need to reach theoretical saturation rather than to identify all eligible studies [ 6 , 30 , 34 , 46 ], might be more appropriate. Systematic reviews of qualitative research inhabit the point at which two research traditions meet [ 38 ]. On the one hand, the methodology of systematic reviews, developed principally over the last two decades, has been dominated by quantitative systematic reviews. Systematic reviews (SRs) of trials attempt to locate every possible study on a given topic or intervention [ 35 ]. Some authors [ 10 , 63 ] advocate a similar approach for QES. On the other hand, Booth [ 54 ] argues that, rather than adopting a “trials-type search,” authors should use a “theory” driven approach, resembling “diversity” or “saturation” sampling approaches used in primary qualitative studies. The interpretive nature of QES suggests the value of methods derived from primary qualitative research, such as the use of theoretical sampling until data saturation is reached [ 54 ]. Whereas in quantitative meta-analysis, omission of a key paper is critical to statistically drawn conclusions; this is not true of a QES which aims to make a conceptual and interpretative contribution. Campbell et al. affirm that “omission of some papers is unlikely to have a dramatic effect on the results” [ 37 ]. The creative tension that results from bringing together these two traditions, labelled as a “dual heritage” [ 38 ], is most clearly seen in the literature that describes how to select an appropriate sample.

Selection of an appropriate sample of participants is fundamental to quantitative and qualitative research [ 121 ]. Similarly, selection of an appropriate sample of papers is essential for a successful evidence synthesis [ 38 ]. In quantitative reviews, this is typically framed in terms of a “comprehensive” sample, by implication a universal sample, to minimise bias and to permit subsequent generalisation. Conceptually, however, a comprehensive sample is problematic as it constrained by the number and type of resources to be searched, the diversity of materials contained within such sources and the time available. Recently, I proposed that “exhaustive” be preferred over “comprehensive” because it conveys the finite nature of resources (e.g. searcher time, money, and access to databases; time to sift). “Exhaustive” is relative to the purpose and type of review rather than referencing some mythical absolute [ 76 ]. However, such debate is clouded by the tendency to use “comprehensive” and “exhaustive” interchangeably. Brunton et al. [ 76 ] observe that “exhaustive searching is improbable,…the obligation on reviewers is to plan a thoughtful and clearly described plan [sic] to locate the sample of studies most likely to answer their research question[s] reliably.” Key difficulties include how to establish the population of potentially relevant studies without identifying all relevant studies [ 45 ]. Data or theoretical “saturation” could have limitations in this context; importantly, how can a reviewer know that an additional study will not add important insights? [ 46 ].

Early commentators expressed anxiety that selective sampling may result in the omission of relevant data, thus limiting understanding of the phenomenon [ 50 , 53 , 60 ]. Gallacher et al. [ 40 ] characterise two schools: those who advocate using purposeful sampling to retrieve materials until data saturation is reached [ 51 ] and those who aim to retrieve all relevant studies in a field rather than a sample of them [ 8 ]. The first approach has logistic and epistemological drivers and is often taken when review teams face a large and diverse set of resources [ 61 ] or when they are developing concepts and theories [ 65 ].

Studies aimed at comprehensively summarising the literature include a comprehensive and rigorous search using predefined index/subject heading/free-text terms, informed by an initial scoping search [ 10 , 30 , 64 ]. Thus, aggregative reviews, characterised by the Joanna Briggs Institute’s proprietorial meta-aggregation method, explicitly seek to mirror the breadth of sources included within an effectiveness review [ 49 ]. Such an approach facilitates comparability between quantitative and qualitative outputs. However, it pays scant recognition to the different epistemological traditions underpinning different types of qualitative synthesis.

For a qualitative reviewer, time is best spent not “piling up examples of the same finding, but in identifying studies that contain new conceptualisations of the phenomena of interest” [ 76 ]. Notwithstanding good methodological justifications for searching comprehensively in SRs of trials, not least to protect against systematic errors such as publication bias [ 76 , 80 ], qualitative data collection is characterised, not by “statistical representativeness” but by “systematic non-probabilistic sampling” [ 122 ]. Several authors [ 52 , 54 , 55 , 65 ] suggest drawing on the sampling techniques of primary qualitative research, including theoretical sampling and theoretical saturation, when synthesising qualitative literature. Booth [ 54 ] states that the intention of QES is not to identify all literature on a particular topic, the aim being identification of papers with characteristics relevant to the phenomenon being studied, not statistical representativeness [ 54 ]. Innovative techniques might be “borrowed” from primary qualitative research such as deliberately seeking studies to act as negative cases, aiming for maximum variability and designing results set to be heterogeneous, as an alternative to “the homogeneity that is often the aim in statistical meta-analyses” [ 32 ].

Downe [ 71 ] describes how theoretical saturation might be operationalised in terms of whether additional studies continue to reinforce the line of argument. Under such circumstances, the author reasons, a search for new studies will reap increasingly diminutive returns, offering justification for truncating the search. O’Connell and Downe [ 73 ] describe how they identified a point of theoretical saturation “when two articles identified late in the search process did not add anything new to the emerging synthesis.” One reviewer reflects that their background as a quantitative systematic reviewer pushed them towards a higher threshold: “I support the ‘data saturation approach’ and think if the next twenty papers don’t offer anything new, what’s the likelihood of the 21 st (reflexive statement)” [ 46 ]. A more mechanical, rather than interpretive, interpretation of saturation relates to the recurrence of studies , rather than themes [ 63 ], as references to the same study begin to reappear repeatedly.

The benchmark of a comprehensive sampling frame persists despite the methodological innovation offered by purposive and theoretical sampling approaches. Adding additional electronic databases to a search protocol (i.e. to search for more of the same) [ 22 ] runs counter to the strategy of seeking to diversify a sample (i.e. purposively to move on to different, more productive lines of inquiry). Several authors comment on the value of identifying the disconfirming case [ 30 , 38 , 54 , 71 ], and search strategies may be targeted specifically to achieve such insights [ 38 ].

A reconciled position would state that the sampling method should be appropriate to the type of review and its purpose. The Centre for Reviews and Dissemination (CRD) guidance recognises the absence of consensus over the issue of sampling [ 8 ]. Rather than predicate their guidance on epistemological concerns, they suggest that if the number of studies is too large to work through, researchers may decide to adopt a strategy for limiting the number of included studies. Purposive and/or theoretical sampling are the main choices, with papers selected for inclusion on the basis of such criteria as rich description or conceptual clarity [ 8 ]. The guidance highlights a role for random sampling, probably most appropriate when constructing a test set for a methodological investigation. Cited examples include purposive sampling derived from qualitative meta-synthesis [ 58 ] and critical interpretive synthesis [ 65 ].

Reconceptualising literature searching for QES, and indeed knowledge synthesis more generally, around the appropriateness of the sample rather than its completeness opens up an exciting variety of sampling approaches derived from qualitative research [ 78 , 79 ]. Patton’s 16 strategies [ 123 ] could ultimately be matched to the full range of synthesis types in an expanded version of Table  3 and then translated into corresponding search techniques for each sampling method.

How many studies are enough?

A further cause for debate relates to the number of studies to include within a QES [ 34 , 71 ]. Some methods of synthesis, such as meta-aggregation and meta-study, make a virtue of being able to handle large numbers of studies. More interpretive approaches privilege smaller numbers of studies [ 12 , 71 ]. While it is undesirable to talk in terms of specific numbers [ 124 ], the amount of relevant data may be a function both of the number of studies and their conceptual richness and contextual thickness [ 8 ]. Furthermore, data considered rich and thick in relation to one aspect of the review question may be scarce in relation to another aspect, even within the same set of studies. Too few studies may limit the support for the entire synthesis or for individual constituent themes. Too many included studies may impair the data analysis, making conceptual analysis “unwieldy” or making it difficult to maintain insight or “sufficient familiarity,” [ 37 ] thereby obscuring patterns that are apparent within a smaller set of studies [ 46 , 59 , 68 ]. In seeking such a balance, we typically arrive at a preferred number of between 6 and 14 studies. Campbell et al. [ 37 ] suggest that “a maximum of about 40 papers is realistic because it is difficult to maintain sufficient familiarity with >40 papers when trying to synthesise them all….” Where the number of studies to be included falls short of the total population of eligible studies, for whatever reason, it becomes critical that “reviewers explicitly and transparently state their criteria for including studies” [ 34 ].

When can I stop searching?

In qualitative research, analysis and data collection occur simultaneously, often to the point where no new ideas are developing [ 46 ]. Thus, it is unlikely that a reviewer can pre-specify a set number of studies without considering their richness, thickness and overall quality. For situations other than exhaustive sampling, a reviewer must develop clear explanations for the circumstances under which searching was terminated. Stopping rules have been proposed, initially for methodological reviews, but these may apply, by extension, to reviews that seek to achieve saturation [ 82 ]. Quantitative reviewers currently seek methods to define a point beyond which further literature searching has little justification [ 30 ]. For qualitative reviews, the answer may lie in the principles of data saturation [ 30 ].

Sampling issues

The use of alternatives to comprehensive searching potentially creates several problems, particularly where a reviewer has chosen to locate their QES within a systematic review paradigm [ 34 ]. Some commentators express concern that alternative sampling approaches open a QES to allegations of subjective decision-making [ 75 ] or assertions that such reviews are no longer transparent or reproducible [ 30 , 34 ]. Others respond that “systematic” should not be misappropriated to favour one research system over an equally legitimate alternative [ 38 ]. Nevertheless, pressure to observe quantitative systematic review conventions persists with 81 % of published meta-ethnographies using exhaustive search strategies [ 42 ]—this for a methodology that, as seen in Table  3 , recognises the appropriateness of purposive sampling approaches. Indeed, the originators of the meta-ethnographic approach caution against exhaustive inclusion of data as it is likely to lead to over-generalisation and “trite conclusions” [ 4 ].

Just as there is no consensus regarding the number of interviews required for a “good” qualitative study, there is no consensus on what type of sample is required for a good qualitative synthesis [ 46 ]. France et al. [ 42 ] identify a need for further exploration of those circumstances under which exhaustive searches are desirable or necessary. Pragmatically, review teams need to bear in mind, when sampling, the underlying theoretical perspective together with a need to be explicit about any strengths and weaknesses of their approach [ 71 ]. This includes a requirement to communicate their sampling approach, methods used and the rationale that underpins the sampling approach.

Finally, while epistemological distinctions may occasion different sampling strategies, this distinction may not be quite so apparent in searching practice. The need to search extensively and to follow up any potentially useful lines of inquiry, while not driven by statistical considerations, may be no less present when seeking to find qualitative studies [ 32 ].

For inclusion in this section, a publication was required to either (i) provide an overview of sources to be used in the context of QES or (ii) include a substantive discussion of the topic of selection of sources within a published synthesis or (iii) mention the selection of sources within any of the works identified for the “ Overviews, summaries and guidance ” section. A total of 22 items was thus included in this section [ 6 , 8 , 14 , 18 , 22 , 34 , 37 , 39 , 45 , 46 , 63 , 83 – 85 , 87 – 94 ].

Coverage of databases

In a sample of QES published in 2012–2013, Wright et al. [ 94 ] reports the number of databases searched per review ranged from 3 to 20, with 37 % searching from 3–5 databases, 28 % searching from 6–8 databases, 14 % searching 9–11 databases and another 14 % searching 12 to 14 databases. Seven per cent of reviews searched over 16 databases. Reliance on MEDLINE alone is particularly discouraged [ 83 ]. However, a meta-ethnographic study in complementary and alternative medicine searched 67 different database sources and yet found 87 % of included qualitative studies from PubMed alone [ 90 ]. CINAHL and DIMDI (a German database) also yielded a high number of relevant hits.

Several authors point to the superiority of CINAHL’s coverage of qualitative research [ 6 , 8 , 14 ]. The Cochrane Handbook [ 6 ] and CRD [ 8 ] highlight that CINAHL introduced “qualitative studies” in 1988, reflecting a particular interest in qualitative studies for nursing researchers while MEDLINE did not add a corresponding subject heading until 15 years later. However, technical efficiency should not be confused with coverage with CINAHL at 4.5 million records covering approximately a quarter of the records included by MEDLINE. In their recent study of multiple QES, Wright et al. [ 94 ] demonstrate that, assuming a rigorous search strategy and accurate indexing, CINAHL is a good source of primary studies with the potential to yield unique studies.

Surveys consistently report MEDLINE and CINAHL as the two most frequently searched sources of qualitative research [ 39 , 84 ]. CRD guidelines [ 8 ] stipulate that “assuming that the topic of interest falls within their scope, then searching both MEDLINE and CINAHL is likely to be important.” CINAHL and MEDLINE retrieved references most relevant to a search on nursing manpower and EMBASE did not provide substantive additional information [ 22 ]. They recommend that both CINAHL and MEDLINE be consulted when planning an optimal bibliographical search related to nursing topics as differences in coverage were striking.

While publication bias possesses a lower profile within qualitative than quantitative research, review authors must be aware that limiting a search to well-known databases may result in missing useful information. In particular, review teams should identify specialist databases that relate to a particular topic and databases that contain particular types of publication, e.g. Dissertation Abstracts and supplementary search strategies that may increase the chance of finding grey literature or of retrieving journals not indexed by the mainstream databases [ 34 ]. McGinn et al. [ 91 ] report the performance of databases across a small set of social care topics; CINAHL performed at a consistently moderate level of sensitivity across topics, and Social Care Online performed consistently poorly. Social Services Abstracts (SSA) was the best performing database [ 91 ] although this again is likely to be topic specific [ 125 ] with certain databases being indicated for certain kinds of questions [ 87 ]. Some databases favour organisational-type questions while others privilege more clinical-type questions. McGinn and colleagues [ 91 ] observe an “unpredictability” around database performance across topics. This occasions researchers to use conservative risk-averse strategies such as consulting greater numbers of databases and screening larger numbers of hits. As in the relatively well-developed area of health care, there is an ongoing need for database comparison case studies across a wider variety of subtopics, thereby building up a body of evidence on retrieval for qualitative research.

National bodies commissioning reviews typically seek strong representation of indigenous studies within the evidence base for a particular review question. Stansfield et al. [ 92 ] demonstrated that a UK-fortified set of seven additional databases (British Education Index, Child data, IBSS, Index of British Theses, Social Care Online, The British Library Integrated Catalogue and Zetoc) yielded additional unique studies. Importantly, they did not only limit themselves to examining retrieval rates but also attempted to assess the impact on findings from the final review. Of five studies identified through UK-fortified strategies, one study was central to development of a descriptive theme while the other four less influential studies added detail and strength to the review’s findings. Furthermore, these studies were of generally high quality, contrasting with the methodological “futility” encountered in a corresponding investigation of effectiveness studies [ 126 ]. This represents an important future direction for evaluation of search sources, strategies and procedures.

Grey literature sources

Grey (or “fugitive”) literature (e.g. technical reports, working papers) is frequently cited as an important source. Commonly, reviewers pay homage to the potential value of searching grey literature and then reject its feasibility, citing limitations of time and costs [ 22 , 37 , 91 ]. In actuality, we know little about the impact of publication bias specifically on qualitative research. We understand that researchers will often want to avoid opening up the prospect of time-consuming and minimally productive follow-up of the unpublished literature. Nevertheless, it is unhelpful for reviewers to imply through terse reporting that they have “taken care of” this uncontrollable mass of alternative publications. McGinn et al. [ 91 ] describe how they pragmatically accepted grey literature, identified serendipitously, when an item satisfied search selection criteria but not specifically searching for it. This may open up a review team to charges of being “unsystematic.” We consider it unhelpful either to plan to search the “grey literature” or to claim to have done so—it is preferable to pre-specify exactly what forms of literature are being sought and then to select sources and strategies for these specific forms, e.g. theses, process evaluations, hospital internal reports, research reports, conference proceedings, studies produced by charities. Some authors have compiled lists of grey literature sources specific to health care [ 85 ] and social care [ 87 , 127 ].

Books and book chapters

Several commentators highlight that qualitative research is published in books as well as journal articles. Strategies for searching books and book chapters require particular consideration [ 34 , 37 , 45 , 63 ]. Some differences relate to the social science disciplinary background being substantively different from the literature of medicine [ 18 , 37 ]. Campbell et al. describe inclusion of books and book chapters in their two meta-ethnography case studies [ 37 ]. They identify the limitation of “truncation bias” in connection with journal articles as the full details of a descriptive qualitative study are unlikely to be published in a short article. Campbell et al. therefore recommend using multiple databases and search strategies in order to maximise the yield of relevant qualitative papers [ 37 ]. Searching for books may be achieved through relevant organisational websites, book catalogues, Google Scholar and consultation with librarians [ 88 , 89 ]. The determining factor is likely to be the resources available to an individual project [ 37 ].

Further potentially useful sources, particularly given that a qualitative research project is typically feasible within the constraints of an academic qualification, are dissertations and theses [ 46 ]. Some authors exhibit a similar resistance to including theses as they do for grey literature in general. Stated reasons for this range from practical considerations, to keep the number of papers manageable [ 37 , 46 ] and to prioritise the literature that is easier and quicker to access [ 37 ], through to concerns about items not being peer reviewed and published reports [ 46 ]. Searching for theses is challenging as they are not indexed in the same way as journal articles and may be accessible only from experts (researchers, providers, policy makers) or via specialist theses databases [ 37 , 93 ]. Access to relevant studies may be achieved by searching relevant organisational websites, Google Scholar, thesis databases, specialist journals and consultation with librarians [ 88 , 89 ]. It may be feasible to include only recent theses as they are less likely to possess published journal counterparts [ 37 ]. A unique methodological issue is the depth of reporting possible in a PhD thesis. If one or two theses are included alongside a larger number of published articles, constrained by word limits, they may “swamp” the data from these naturally thinner studies. It is preferable, where possible, to identify published journal articles derived from theses, thereby making the units of analysis more readily comparable. Nevertheless, the volume of data from more extensively reported theses is not an argument against their inclusion per se, simply against the uncritical use of theses, making procedures of quality assessment correspondingly more critical.

Structured questions

To be included in this section, a publication should (i) provide an overview of structuring of review questions within the context of QES or (ii) include a substantive discussion of structuring review questions within a published synthesis or (iii) any relevant mention in works identified for the “ Overviews, summaries and guidance ” section. A total of 17 items was included in this section [ 9 , 10 , 13 , 15 , 28 , 30 , 31 , 34 , 41 , 65 , 73 , 95 – 98 , 101 , 102 ].

The literature on qualitative searching tends to reflect four approaches to use of a structured, formulated review question. A minority of commentators assume that the requirement to formulate the question as a Population-Intervention-Comparison-Outcome (PICO) is shared across quantitative and qualitative review types [ 101 ]. The PICO format is underpinned by assumptions derived from epidemiological study design seen in the terminology used. A large majority of variants propose modifications that reinterpret the PICO approach, e.g. 3WH [ 28 ], PEICO(S), PICo, PICOC, PICOS, SPICE, SPIDER [ 98 ]. Several commentators suggest specific question formulations for specific purposes, e.g. CIMO [ 96 ], ECLIPSe [ 95 ] BeHEMoTh [ 9 ]. Noticeable among these second and third approaches are the typical addition of elements capturing context (aka environment or setting, e.g. BeHEMoTh, CIMO, PEICO(S), PICo, PICOC, SPICE) and the stance of the affected party (e.g., perspective in SPICE and stakeholders in PEICO(S)) (Table  4 ).

A significant few question the appropriateness of a pre-specified (i.e. a priori) question at all. Drawing upon the primary qualitative heritage of grounded theory, they assert that the review question only emerges from a preliminary analysis of the data. Related to this issue are two other considerations; first, whether iterative approaches are appropriate to searching for qualitative research and second, when a QES accompanies an SR of trials, whether the scope of the review questions should be coterminous.

As with quantitative reviews, there is little empirical data to support the merits of question formulation [ 13 , 101 ]. With regard to the choice of specific frameworks, limited evidence suggests that PICO may be preferred when the primary objective is sensitivity whereas SPIDER favours specificity [ 101 ]. The authors recommend a modified PICO with added qualitative search terms (PICOS) which optimises the trade-off between sensitivity (not missing relevant items) and specificity (only retrieving relevant items) for circumstances when a fully comprehensive search is not feasible.

Differences between quantitative and qualitative review questions

Lorenc et al. [ 41 ] describe circumstances where the qualitative review question may not directly mirror the PICO of an SR of trials. In their review of preventive interventions for skin cancer, they identified useful non-interventional studies that were about attitudes to sun behaviours or skin cancer in general [ 41 ]. They reinterpreted their inclusion criteria to include any study reporting qualitative evidence relating to sun protection beliefs or behaviours, regardless of a link to a specific intervention. They conclude that this finding might translate to public health, and social and health research more generally, where relatively little qualitative evidence on specific interventions is available. Data linked to specific interventions were not necessarily of greater value than data related to broader attitudes.

Where QES are conducted in parallel with SRs of trials, review teams may have to adopt different conceptual schemata for their inclusion criteria and search strategies [ 41 ]. In contrast to current guidance, that seeks a common question structure (e.g. PICO) for the SR of trials and the QES, the authors flag that “structural divergence” between the two questions may be inevitable [ 41 ].

O’Connell and Downe [ 73 ] attempt to reconcile the tension between the need to preserve flexibility and yet “maximise rigour” through an explicit two-stage process. This process involved iteration in regard to topic definition followed by tight control over inclusion and exclusion, study quality and analysis. Scoping a topic, primarily performed for logistic considerations in an SR of trials, becomes correspondingly more important if the review team is to ensure “secure” concepts within the context of a QES. Divergence between commentators on the need for a pre-specified formulated question is partly explained by whether they consider the scoping process to be preliminary to, or integral to, the review process. So Ring et al. [ 34 ] state that QES typically start with a relatively well-defined research question and yet acknowledge, with Dixon-Woods et al. [ 30 ] that, according to their philosophical approach, some QES reviewers modify their initial research question in response to literature searching and screening [ 34 ].

Some commentators make a useful distinction between summative or aggregative QES, where research questions are generally established a priori and relevant research reports are identified exhaustively, and knowledge-building or theory-generating QES where such pre-specification may inhibit creativity [ 15 ]. The latter, they argue, starts from a less-clearly defined focus and evolve iteratively [ 10 ]. Within such a context, the expansive, as opposed to exhaustive, literature search can be viewed as a creative vehicle for continually redefining the research question and exploring the emergence of research findings. Consequently, Dixon-Woods and colleagues [ 65 ] evoke earlier qualitative researchers in suggesting that this process, with the review question being highly iterative and modified in response to search results, treats the question as a compass rather than an anchor.

Search procedures

Publications in this section should either (i) provide an overview of topic-based database search procedures within the context of QES or (ii) include a substantive discussion of topic-based database search procedures within a published synthesis or (iii) mention database search procedures within works identified for the “ Overviews, summaries and guidance ” section. Six items were thus included in this section [ 15 , 21 , 34 , 35 , 37 , 103 ].

A literature searcher faces a dual challenge in how best to optimise the trade-off between recall and precision, thereby keeping the expenditure of resources within manageable limits [ 37 ]. Several variables determine appropriate search procedures. These include how diffuse or broad the topic for review or synthesis is which requires a wider net and inclusion of more databases [ 37 ]. Exhaustive searches often necessitate “trawling” to identify every possible study. They often prove time-consuming and result in large numbers of non-relevant studies [ 34 ]. Strategies that attempt to maximise the number of potentially relevant records (high sensitivity) often result in a large number of non-relevant studies (low specificity) [ 21 ]. A review team should seek to optimise the ratio between the number of relevant references and the number of retrieved references (the “hit rate”) for sensitive topic-based searches and reflect whether available time might be better spent conducting citation searches on the Web of Science or Google Scholar (see the “ Supplementary strategies ” section). Finfgeld-Connett and Johnson [ 15 ] distinguish between extensive search approaches (that map to exhaustive searches) and expansive approaches (which progressively explore emerging lines of inquiry).

Mackay [ 103 ] differentiates between “qualitative” and “quantitative” searching approaches highlighting a similarity to the difference between qualitative and quantitative research methods. A quantitative searching approach is linear and structured based on objective and reproducible identification of pre-specified literature. Qualitative search approaches are concerned with the essential and peculiar character of phenomena and recognise that searching is never value free. Qualitative searching strategies are slow, labour intensive and difficult to replicate (because of the amount of time needed). They may be used when a topic is not dominant in the discourse of the literature and/or the topic is not well-conceptualised in the literature. The comprehensive a priori quantitative search contrasts with the intuitive and recursive follow-up of the purposive, iterative qualitative search. Both strategies can be systematic or not depending on how disciplined the searcher is. Documentation of the a priori (protocol-driven) search strategy (e.g. using screen captures) ensures that the search is explicit and thorough. For the iterative searching approach, a process analogous to memoing may be used to record the working notes of the searcher. Thus, both qualitative (iterative) and quantitative (a priori) search approaches can be systematic if the searcher is explicit about their searching processes.

Even though all synthesis methods include iteration, the degree, and the review stage at which iteration takes place, varies. Framework synthesis and critical interpretive synthesis explicitly involve iterative literature searching while realist synthesis and meta-narrative involve iteration at every stage [ 35 ]. Several synthesis methods do not explicitly mention iterative searching and thus implicitly subject themselves to a priori and positivist assumptions [ 35 ]. Meta-aggregation follows closely the single pass a priori - formulated search strategy model, the “big bang approach” which relies upon pre-identification of searching strategies, inclusion/exclusion criteria etcetera and implementing these with fidelity [ 35 ]. Increasing awareness of the array of sampling methods available and appropriate for synthesis coupled with the pragmatic demands of conducting reviews in public health or social work practice is likely to result in wider uptake of iterative searching. However, iterative searching poses significant challenges to the reporting of search strategies and may well subvert the discipline imposed by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram [ 128 ].

The process of development of a search strategy for a QES is not demonstrably different from that for an SR of trials [ 34 ]. In both cases, searches need to be developed for the topic area and, separately, for the types of studies to be included [ 34 ]. The searcher must judge the optimal balance between sensitivity (not missing relevant items) and specificity (only retrieving relevant items) for both the topic and the study type. A very specific interpretation of qualitative research might involve only searching for words relating to ethnographies. A sensitive interpretation might involve specifying the types of phenomenon (e.g. views, attitudes, feelings), the study types (e.g. phenomenology), the data collection methods (e.g. interviews, questionnaires and focus groups) and the types of data (stories, narratives, etcetera).

A related issue concerns the type of data source to be included within the QES. The Cochrane Qualitative and Implementation Methods Group concurs with other review teams (e.g. [ 10 , 28 , 37 ]) by operationalising “qualitative research” as research using a recognised method of qualitative data collection and a recognised method of qualitative data analysis. More forgiving interpretations might include data from open-ended responses to questions in an otherwise quantitative questionnaire or survey. A narrow set of studies might be retrieved by only synthesising qualitative data reported in or alongside randomised controlled trials (e.g. from pilot studies, feasibility studies, process evaluations). We coined the concept “sibling studies” to characterise studies that derive from the same parent study but that report a particular slice of the data [ 7 , 110 ]. Glenton et al. [ 104 ] explored the use of directly related (sibling) qualitative studies in connection with a Cochrane review of lay health workers. Only a small proportion of included trials had carried out some form of qualitative data collection during or after the intervention. Data were “sparse” with methods and results being poorly described. Their findings echo an earlier study by members of the same team [ 105 ] that found only 30 of 100 trials had associated qualitative work. Furthermore, around half of these sibling studies pre-dated publication of the trial [ 104 ].

Reviewers may also decide either to include or exclude mixed-methods studies and may choose either to synthesise such studies in their entirety or to focus only on the qualitative component of the larger study [ 8 ]. However, CRD guidance cautions against relying on strategies designed to retrieve clinical trials as a route to identifying qualitative associated or linked counterparts, citing the Cochrane Handbook requirement for structured searching [ 129 ].

The scope of an SR of clinical trials may differ from that for an accompanying QES [ 41 ]. Qualitative researchers may not have conducted research around a particular intervention, particularly where it is novel or experimental. A review team may need to access research about the patient’s experience of their condition, barriers and facilitators for existing treatments, and the characteristics of an “ideal” intervention to address review questions relating to feasibility and acceptability. Furthermore, few primary studies are likely to share the same research question or focus as the planned synthesis. However, this does not mean that these studies may not yield relevant data [ 34 ]. “Dropping” the intervention concept makes the search strategy broader for the qualitative component than for its quantitative counterpart. Alternatively, for interventions where context is important (e.g. cultural attitudes or health service specific effects), a qualitative synthesis may implement a narrower interpretation of scope, such as countries with a comparable health system (to facilitate transferability) compared with the SR of clinical trials (which aspires to generalisability). The Cochrane Handbook [ 6 ] cautions that seeking to retrieve qualitative studies from a topic-based search strategy designed to identify trials is methodologically unsound. A trial-based search strategy is not designed to identify qualitative studies. Indeed, the trial-based strategy may well achieve a measure of specificity by purposefully excluding many qualitative research types.

Where the scope of quantitative and qualitative reviews is co-terminous a review team can employ a broad approach using subject and topic terms without specifying the study type(s) of interest [ 8 ]. Both quantitative and qualitative studies would be identified. However, this method generates large numbers of retrieved records and requires those sifting the abstracts to be equally adept at identifying both types of study. Such an approach is employed at the EPPI-Centre where they routinely conduct reviews of “views studies” alongside reviews of effectiveness [ 130 ].

In reviewing the quality of reported search procedures in a sample of published qualitative syntheses, Dixon-Woods et al. observe that search techniques often lack sophistication and are thus likely to miss relevant material [ 88 ]. They suggest a need to involve an information specialist in the search process, already well-recognised for quantitative systematic reviews.

Search strategies and filters

For inclusion in this section, a publication was required to either (i) provide an overview of search strategies and/or methodological filters within the context of QES or (ii) include a substantive discussion of search strategies and/or methodological filters within a published QES or (iii) any mention of search strategies and/or methodological filters within works identified for the “ Overviews, summaries and guidance ” section. Sixteen items were included in this section [ 8 , 16 , 17 , 19 – 25 , 34 , 37 , 44 , 77 , 88 , 90 ].

The development of pre-specified search strategies using methodological terms has an extensive pedigree within quantitative research, particularly for randomised controlled trials. Methodological “filters” or “hedges” are specially designed search strategies used to retrieve citations of clinically relevant and scientifically sound studies (or reviews) [ 131 ]. These search terms are initially suggested by librarians and clinical users, and then, performance metrics are generated for these terms both singly and in combination. The performance of hedges for clinical trials was subsequently enhanced by a Cochrane-associated retrospective indexing initiative. Almost a decade later, the Hedges Project at McMaster University expanded its battery of empirically tested methodological filters to include qualitative research filters for the four principal health-related databases, namely MEDLINE [ 25 ], CINAHL [ 24 ], PsycINFO [ 19 ] and EMBASE [ 23 ]. A range of filters (sensitive—to minimise the potential of missing relevant references, specific—to minimise the potential of including irrelevant references and optimal—to determine an efficient trade-off between sensitivity and specificity) is available for each database (Table  5 ), supported by information about how the filters were developed to help in selecting an appropriate filter ( http://hiru.mcmaster.ca/hiru/HIRU_Hedges_home.aspx ). A survey of members of the Cochrane Qualitative Methods Network revealed some resistance to the use of filters with associated concerns about whether filters were suitably comprehensive [ 17 ]. Notwithstanding these concerns, qualitative filters demonstrate a performance that compares favourably with that for retrieval of trials. More recently, the InterTASC Information Specialists’ Sub-Group (ISSG) of information professionals has produced a Search Filter Resource ( www.york.ac.uk/inst/crd/intertasc/ ) documenting the evidence base for published filters.

The performance of filters is liable to change over time [ 22 , 125 ], with new terms being added by the user community, changes to indexing terminology and journal coverage and the appearance of specialist qualitative journals. The utility of the MEDLINE empirically tested filter [ 25 ] was compromised, albeit to a minor degree, by the appearance of the MeSH term “qualitative research” in 2002, subsequent to creating the test and validation sets. Differences exist in the indexing of qualitative research within electronic databases such as MEDLINE, EMBASE, PsycINFO and CINAHL [ 77 , 88 ]. Reviewers should avoid running a filter devised on one database against another database or, less obviously, translating terms on a one-by-one basis from one database to another. In the context of research syntheses, more generally, Cooper [ 132 ] encourages searching multiple databases simultaneously where possible, to avoid excessive duplication. However, a simultaneous searching of multiple databases requires that the searcher develop a strategy that is not overly reliant on specific indexing terms, particularly as they might be artificially inverted (e.g. education, professional) and thus retrieve zero hits on databases other than their parent source.

Optimally, a sensitive strategy retrieves individual terms regardless of how they occur within indexing languages although in the example of “Professional AND Education” the numbers of retrieved results and the high proportion of “false hits” would prove prohibitive. It may be useful to undertake a content analysis of the subject headings assigned in each database to determine why indexing terms diverge across databases [ 22 ]. The more varied the databases, the more challenging creation of a “standard” search filter for application to all databases becomes, suggesting that search strings need to be “adapted to the idiosyncrasies of each synthesis to achieve the best results” [ 37 ]. Generally, methodological filters for qualitative research have undergone little replication and validation [ 16 , 34 , 88 ]. It is not known whether search filters, as developed for health care [ 20 , 23 – 25 ], are equally feasible and useful within disciplines such as social work or education.

In the context of filters in general, Jenkins [ 106 ] attempts to identify different generations of search filters with corresponding degrees of rigour. Foremost among these are filters developed from a single “gold standard” set of unequivocally relevant references (i.e. identifiable qualitative research studies) and then randomly splitting these records into a development and validation set. Unfortunately, few available filters or hedges for qualitative research meet this empirical standard.

A search strategy to be effective requires (i) that it retrieves relevant records, (ii) that it does not retrieve irrelevant references and (iii) that the collective terms be parsimonious, thereby avoiding redundancy. The third requirement is a particular current concern for the qualitative searching community. Lengthy filters have been devised, collecting together research methodology terms or all possible terms to capture “patient views.” However, once a particular record is retrieved by one term, there is little value in retrieving it again using subsequent terms. A reviewer is primarily interested in retrieving additional different records or in retrieving a record on a subsequent database if it has been missed because of the indexing particularities of a previously searched database [ 77 ]. Two case studies [ 16 , 44 ] suggest that a parsimonious strategy involving the terms qualitative, findings and interview* (as both text word and index term) may perform acceptably well when conducting a search for qualitative research across a range of databases. Such a broad free-text strategy performed particularly well on the CINAHL database [ 8 ]. It is critical that this finding is tested across multiple topics and time spans.

Searching with “broad terms” such as “qualitative research,” “qualitative studies” and “interview,” together with their topic terms, may be equally applicable for all databases [ 90 ]. In contrast, specific MeSH terms or methodological index terms that aid the identification of qualitative research (“exploratory,” “grounded theory,” “content analysis,” “focus groups” and “ethnography”) provided no additional results and delivered numerous false hits. An evaluation by Shaw et al. [ 21 ] confirms that an optimal strategy for retrieving qualitative research will combine specific free-text terms, broad terms and thesaurus terms; relying on one strategy alone would fail to identify relevant records. This evaluation identified low precision for all three types of strategy, with only 4 % of papers proving to be relevant at full-text screening. Campbell et al. [ 37 ] describe using the single term “qualitative” for title searches of the ZETOC contents database. Anecdotally, searches of Google Scholar enjoyed some success by adapting the MEDLINE-based “qualitative OR findings OR interview*” strategy, either combined with subject terms or within the “Search within Cited Articles” feature for already-identified relevant citations.

Supplementary strategies

To be included, a publication was required to either (i) provide an overview of supplementary (i.e. non-topic-based) search strategies within the context of QES or (ii) include a substantive discussion of supplementary (i.e. non-topic-based) search strategies within a published QES or (iii) include any mention of supplementary search strategies within works identified for the “ Overviews, summaries and guidance ” section. A total of 24 items was thus included in this section [ 8 , 10 , 14 , 20 , 30 , 34 , 37 , 39 , 42 , 45 , 62 , 63 , 84 , 94 , 107 – 116 ].

Guidance on meta-ethnography advocates that, in order to minimise the risk of missing studies, searchers conduct supplementary searches alongside topic-based database searching [ 37 ]. The authors propose a multipronged approach; hand-searching relevant journals; contacting experts in the field of enquiry for curricula vitae and information; and examination of the “grey literature,” conference proceedings etcetera [ 37 ]. They rightly advise that decisions should be made at each stage depending on the resources available. Search strategies for qualitative research should extend beyond electronic databases [ 20 , 30 , 63 ] but knowledge about which strategies to use for particular topics is not forthcoming [ 109 ]. Several commentators report that supplementary search strategies are useful in compensating for deficiencies of indexing terms and the limited value of “protocol-driven search strategies” [ 8 , 10 , 14 , 62 ]. Key journals are hand-searched in case electronic searches are not sufficiently sensitive or where indexers have not assigned adequate keywords [ 37 ].

Noticeable is a trend to favour creative approaches to retrieval—most notably Bates’ berry picking approach [ 107 ]. Barroso et al. [ 10 ] used berry picking as a framework for their search techniques for a meta-synthesis project on women with HIV infection. Combining formal search strategy methods with berry picking may help to expand searching from a broad topic towards “new, unpredictable ideas and directions” and even reformulation of the original query [ 108 ]. More recently, berry picking has been revisited, as an alternative to extensive keyword-based approaches [ 15 , 40 ].

Booth et al. [ 110 ] have designed a search approach that seeks to place Bates’ berry picking [ 107 ] on a more systematic footing. The CLUSTER approach seeks to maximise identification of associated or linked studies, identifying not only studies that are instrumentally linked as “sibling studies” but also studies that are theoretically or conceptually associated, “kinship studies” [ 110 ]. From a single “key pearl citation,” the authors conduct searches to find contextually or theoretically proximate documents. They follow up Citations, trace Lead authors, identify Unpublished materials, search Google Scholar, track Theories, undertake ancestry searching for Early examples and follow up Related projects (embodied in the CLUSTER mnemonic) [ 110 ].

Increasingly, reviews of complex interventions require that a review team identifies the conceptual underpinnings and explores the contextual detail. The CLUSTER method aims to retrieve both concepts and contexts [ 110 ]. While not all commentators acknowledge that theory can be retrieved in a systematic way, Booth and Carroll [ 9 ] have recently devised a structured procedure, BeHEMoTh, of steps to retrieve papers reporting theory.

Dixon-Woods et al. [ 84 ] reported that the most common supplementary strategies used alongside bibliographic databases were following up reference lists and hand searching. Subsequently, Hannes et al. [ 39 ] found that reference or citation searching was used in more than half the QES in their sample. Other popular search strategies included hand-searching journals, contacting experts or authors or web searching. Reviewers also mentioned personal correspondence, related paper options in existing databases, email discussion lists, footnote chasing, or searching conference abstracts, etc. Other approaches include scanning conference proceedings, contacting professional bodies, searching for grey literature and looking at included studies of earlier reviews, personal correspondence, related paper options in existing databases, email discussion lists, footnote chasing or searching conference abstracts [ 39 , 42 ].

Greenhalgh and Peacock, frequently cited in support of deficiencies of topic-based search strategies, report an audit of sources for a review of complex interventions, of which a proportion relates to qualitative evidence [ 62 ]. Only 30 % of included studies were identified from databases and hand searches. About half of studies were identified by “snowballing” (e.g. reference, footnote and citation tracking) and another 24 % by personal knowledge or personal contact. However, the team had recognised a priori that their topic area was diffuse and ill-suited to keyword-based strategies (in essence becoming self-fulfilling) and report relative percentages where increased effort aimed at one source, e.g. personal contact, makes the remaining sources appear less useful. A rigorous evaluation would study whether items could have been identified using databases, regardless of how they were actually identified . In another case study, citation searching, reference checking and contact with experts yielded 11 of 41 included studies [ 20 ]. The use of citation pearl growing (i.e. using known relevant items to identify supplementary search terms) was of limited value because none of the 10 candidate databases, from which the other 31 included studies were derived, included more than four relevant items and therefore did not offer sufficient data for analysis.

Reference checking

Gomersall et al. [ 115 ] report that using reference lists of relevant literature identified 38 relevant articles. Critical, however, is how many included studies were identified uniquely from reference lists. Similarly, Steen et al. [ 116 ] found that backchaining (i.e. checking of reference lists from included studies) identified a further six studies of potential relevance. However, their paper is unclear on how many of these were uniquely identified and subsequently included in the review. Malpass et al. [ 114 ] checked reference lists and contacted authors unearthing one relevant paper which was of sufficient quality to be included in the synthesis. Checking the context of a citation within the paper, not just its appearance in a reference list, is particularly helpful when titles are not informative [ 34 ]. There is a compelling argument to suggest that checking of references in the full-text of already included, or indeed potentially includable retrieved studies, should not be regarded as a supplementary technique but rather simply as standard good practice.

Hand searching

Several reports rate hand searching of relevant journals as “useful” [ 109 ]. However, such anecdotes usually lack data on yields or time spent hand searching. Hand searching is particularly indicated where relevant data is “buried” within the text of a paper and the study is not retrieved through electronic searches [ 34 ]. Typically, hand searching is a misnomer as browsing of titles and abstracts is facilitated online or, where available, a journal’s search facility offers full-text searching over and above the title and abstract facility offered by most bibliographic databases. Additional time should be allowed for supplementary activities [ 34 ]. In published audits, Qualitative Health Research [ 84 ] and Journal of Advanced Nursing [ 39 ] were the most common outlets for QES. For meta-ethnographies, France et al. [ 42 ] reported that the majority (41 %) were published in nursing or midwifery journals, a higher proportion than identified in an earlier audit (32 %) [ 39 ]. Such data may however be confounded by the reported superiority of CINAHL when indexing qualitative publications.

Citation checking

Citation checking harnesses the degree of “relatedness” between an original source and its citing paper. However, “related” items may share a topic, methodology or some tangential or obscure connection. Citation checking (forward chaining) may variously and unpredictably perform better or worse when compared to a keyword-based subject search. Atkins et al. [ 45 ] found citation searching of limited use locating only three of 44 included studies through this method and consulting with experts combined. Even though some review teams report limited success from citation checking, they may differ in their thoroughness or the extent that they use complementary search strategies and so citation checking should not be ruled out.

Contact with authors/experts

The use of experts as a source of potentially relevant citations has received mixed verdicts, from being vital [ 62 , 63 ] through to simply useful [ 111 – 114 ]. Campbell et al. [ 37 ] describe the “striking” importance of consultation with experts, alongside hand searching. As mentioned above, Atkins et al. [ 45 ] found consulting with experts of limited use. Again, the performance of supplementary strategies is relative and depends upon what they are compared with. Greenhalgh and Peacock [ 62 ] reported that contacts with experts yielded significant suggestions of potentially relevant reports when reviewing service-level innovations in health care organisations. Pearson et al. reported that contacting authors of included reports was not an effective use of time or resources. Contact with authors yielded 13 potential leads of which only one poor-quality report was included [ 109 ]. In part, these differences may be topic-specific, but they may equally reflect how good a review team’s networks are and thus represent a source of potential bias [ 109 ].

Other methods

Pearson et al. [ 109 ] identify a further targeted search strategy using programme names for particular initiatives. This supplementary technique is one component of the CLUSTER methodology [ 110 ]. Pearson et al. [ 109 ] conclude that it is unclear from published reports why certain topics or supplementary approaches yield more positive results than others. Rather than endorsing one approach over another, this evidence illustrates the challenges of searching across topics that are poorly defined by database keywords. Retrospective analyses of different search approaches require more detailed reporting than presently available [ 94 ]. For the moment, a review team must judge how they should allocate their overall search resource between topic-based searching and other approaches for their specific topic. However, a multiple search strategy is more likely to identify relevant qualitative research than one relying solely on electronic searching [ 37 ].

For inclusion, a publication should either (i) provide a standard for reporting literature searches within the context of QES or (ii) include standards for reporting of literature searching within a wider reporting standard or (iii) any mention of reporting or documentation of search strategies for QES within works identified for the “ Overviews, summaries and guidance ” section. Seventeen items were included in this section [ 6 , 8 , 12 , 14 , 15 , 21 , 39 , 42 , 45 , 46 , 54 , 70 , 84 , 88 , 89 , 101 , 117 ].

There remains a high degree of consensus that QES should be systematic, explicit and reproducible [ 8 , 54 , 89 ]. Weed [ 70 ], creator of meta-interpretation, observes that the audit trail serves not to enable “member checking” but to make the search transparent and “demonstrate the ‘reasonableness’ of the analysis.” Weed [ 70 ] subsequently suggests a level of detail in QES reports that should include the extent of theoretical sampling and how and why and on what basis studies have been chosen for inclusion in each iteration. He further advocates that complete reporting should include processes by which studies are subsequently excluded, reasons for their exclusion; an interim analysis at the end of each iteration and processes by which concepts for further theoretical sampling has been identified.

Audits of reports of published qualitative evidence syntheses reveal disappointingly low standards of reporting of search processes [ 12 , 39 , 84 ]. Typically, neither the search strategy nor the databases searched are detailed in the published report [ 84 ]. Similar limitations in reporting of search strategies were observed in a recent survey of meta-ethnographies [ 42 ]. Both CRD [ 8 ] and Cochrane [ 6 ] champion the importance of reporting standards for search methods, including documenting the methods for sampling. They highlight proposed Standards for Reporting Literature searches (STARLITE) as a useful resource [ 12 ].

Reporting of systematic reviews is prescribed by the PRISMA, formerly QUOROM, statement. In an attempt to mirror this approach within QES, an international collaboration has produced a tentative reporting standard. The ENhancing Transparency in REporting the synthesis of Qualitative research (ENTREQ) draft statement [ 88 ] recommends the use of the PRISMA flowchart [ 128 ] for reporting, searching, screening and identifying studies for inclusion in the QES. ENTREQ is influenced by STARLITE [ 12 ] in recognising the need to specify the sampling strategy (item 3)—a feature not typically included when searches are comprehensive by default. Table  6 presents the four items from ENTREQ that specifically relate to literature searching and maps these to the elements of the STARLITE mnemonic. However, uptake of ENTREQ [ 88 ] is low. A recent review of meta-ethnography reporting [ 42 ] observed that only one of 19 papers published since ENTREQ’s publication had used the proposed standard to guide its reporting.

Several authors have identified a corresponding need to improve the quality of reporting of primary qualitative research, many focusing on the utility of structured abstracts [ 45 ]. Although this review emphasises potential differences with searching for an SR of clinical trials, both types of review benefit from attempts to improve reporting of search methods. Thus, Kable et al. [ 117 ] provide a 12-step general strategy for documenting the search process for a literature review, heavily informed by qualitative work [ 12 , 14 , 21 , 101 ]. Niederstadt and Droste [ 133 ] specify requirements for reporting and presenting information retrieval processes for health technology assessment, and these too inform presentation of search results for a QES.

Finally, we remark upon a paradox, now recognised in the QES community, that iterative approaches using innovative, yet appropriate, sampling techniques may reflect more informed sophisticated and topic-specific approaches to searching and yet be correspondingly more difficult to report. Reviewers face a choice between a simple, yet easily reported, strategy and a complex, “messy” but accurate, strategy that is more difficult to describe and present. Reviewers should make literature search processes as transparent as possible, even when complex [ 15 ]. However, little practical guidance exists on how to achieve such transparency. France et al. [ 42 ] speculate that one reason why comprehensive searches persist may be attributed to the dominance of established methods and guidance for conducting and reporting quantitative reviews of trials [ 8 , 128 , 129 ]. Toye and colleagues [ 46 ] reflect on how the shaping influence of the Cochrane Collaboration impacted on their decisions to conduct and report their QES and thus satisfy external expectations for rigour.

While there appears to be considerable consensus in relation to the methodology of searching for qualitative research, the findings from this methodological overview should be treated with caution. A limited number of authors and teams are particularly influential within this narrow, specialist area of information retrieval. Furthermore, much accepted opinion can be attributed to the fact that different commentators are drawing upon and citing the same limited set of references in support of their opinions. The methodological guidance is particularly derivative from a few key influential works. Furthermore, retrospective analyses limit the extent to which investigators can take account of search strategy design, database interface and accuracy of database indexing [ 94 ]. Future analyses should collect data on search strategy design, database interface and indexing prospectively to allow review teams to consider the impact of these factors on overall search performance. Finally, the review reveals a paucity of empirical data. Much guidance is based on personal or organisational experience, limited case studies or overworked, and occasionally misapplied, empirical studies, for example the implication that the specific emphasis of search approaches appropriate to a meta-narrative on diffusion of innovations is generically transferable to all QES topics [ 62 ]. Table  7 tentatively suggests some starting principles, with reference to the 7S sections of this review, to inform guidance irrespective of the review producer. On a positive note, we can detect increasing transparency in search methods. Hannes et al. [ 3 ] compared published data from 2005 to 2008 [ 39 ] with data from 1988 to 2004 [ 84 ]. Considerably, more QES papers described the databases they had searched, more reported supplementary search strategies and more chose to specify their search terms.

Towards a research agenda

McGinn et al. [ 91 ] recommend that review teams partner with librarians or information specialists to share the outcomes of case studies that showcase thorough searches and examine their yield. There is a particular need to report data, either within reviews themselves or in subsequent retrospective methodological studies, on where included studies could have been found as well as how they actually were found. The caveat is that reviewers report considerable differences in yields from different sources for different topics (e.g. even for two meta-ethnography case studies by the same team [ 37 ]). Consequently, review teams cannot predict whether topics are more likely to be similar or different, with what has worked previously not being a guarantor of subsequent success in a different topic [ 37 ]. Indeed, the degree to which past performance is a predictor of future performance is currently unknown [ 94 ].

A review of the methodological guidance reveals a need to balance development of generic guidance with development of guidance specific to particular methods of synthesis. The development of specific guidance, where methods of sampling, searching and synthesis are all aligned, is a potential route for reconciling the comprehensive versus purposive sampling debate. Important developments for QES reporting standards are the National Institute for Health Research funded Realist And Meta-narrative Evidence Syntheses: Evolving Standards (RAMESES) project for meta-narrative [ 118 ] and realist reviews [ 119 ] and the Meta-Ethnography Reporting Guideline (eMERGe) project for meta-ethnographies [ 120 ].

Table  8 compiles a research agenda with reference to the “7S” framework of this review. This review possesses several limitations. For inclusion, references must include terms specifically related to searching or retrieval in their titles or abstracts, or cite a limited number of key texts, or be referred to from previously identified items. It is increasingly prohibitive to examine the full-text of all papers reporting QES. Individual reviews may explore innovative methods of information retrieval but not showcase their methodology. However, it is unlikely that this review has completely overlooked important issues given the extent of included articles, studies and guidance. While patterns of co-citation or theoretical saturation are largely unexplored within a methodological context, the reviewer reached a point where no additional items were being identified. Some key items were purposely excluded because they did not differentiate between quantitative and qualitative searching approaches. However, these might be useful, particularly where a mixed methods review is being conducted [ 134 – 138 ].

With regard to the quality of the identified papers, this review detected a high proportion of overviews, occasionally based on or supported by one or more case studies. Indeed, case studies were the most prevalent method used to advance observations on search methodology, with the corresponding weakness that lessons from individual case studies may not be transferable. The shortage of comparative designs or validation studies is likely to be indicative of a corresponding dearth of funded projects exploring methodological aspects of searching for qualitative studies. Similarly, outside the case study evidence base, wider insights derive either from opportunistic samples or from analysing convenience samples of published QES. Notwithstanding the fact that many guidance documents exist in this domain, these too are limited by the weak quality of the evidence used to underpin published recommendations.

QES is an exciting and rapidly developing methodological field, evidenced by a proliferation of methods and of published examples. Decisions regarding search strategy and screening hinge upon such considerations as the review aims, resources, availability of studies and epistemological viewpoint [ 46 ]. However, the popularity of QES should not mask the poor empirical base that exists for many decisions within the searching process. Methodological overviews are largely populated by common empirical studies which are frequently referenced as authoritative. As with quantitative reviews, there is little empirical data to support the merits of question formulation [ 101 ]. Yields from particular databases appear to be largely review specific. Empirical research is required to examine suggestions in the literature that thorough searching of a small number of databases [ 16 , 22 , 90 ], supplemented by other searching methods, may be more efficient than searching across a wider range of databases. We are beginning to learn the merits of different sampling approaches and their alignment to named qualitative synthesis methodologies [ 38 ]. Limited but important evidence exists to suggest that a few qualitative methodology keywords may perform equally well to more extensive filter terms [ 8 , 16 , 17 , 44 ]. Strategies for retrieving books and theses need to be specified with specific agendas remaining to be advanced in terms of searching for process evaluations or mixed-methods studies. Finally, progress has been made in reporting QES, but these standards have neither been validated in the appropriate community nor extended to cover a broad range of QES methodologies [ 88 ]. The QES search methodology research agenda remains ripe for harvesting.

Noyes J. Never mind the qualitative feel the depth! The evolving role of qualitative research in Cochrane intervention reviews. J Res Nurs. 2010;15:525.

Article   Google Scholar  

Petticrew M. Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’. Syst Rev. 2015;4:36.

Article   PubMed   PubMed Central   Google Scholar  

Hannes K, Booth A, Harris J, Noyes J. Celebrating methodological challenges and changes: reflecting on the emergence and importance of the role of qualitative evidence in Cochrane reviews. Syst Rev. 2013;2:84.

Noblit GW, Hare RD. Meta-ethnography: synthesizing qualitative studies. Newbury Park: Sage; 1988.

Google Scholar  

Gülmezoglu AM, Chandler J, Shepperd S, Pantoja T. Reviews of qualitative evidence: a new milestone for Cochrane. Cochrane Database Syst Rev. 2013;11, ED000073.

PubMed   Google Scholar  

Noyes J, Popay J, Pearson A, Hannes K, Booth A. Chapter 20: Qualitative research and Cochrane reviews. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.0.1 [updated September 2008]. The Cochrane Collaboration, 2008. Available from  http://handbook.cochrane.org/chapter_20/20_qualitative_research_and_cochrane_reviews.htm .

Booth A. Chapter 3: Searching for Studies. In: Noyes J, Booth A, Hannes K, Harden A, Harris J, Lewin S, Lockwood C (editors), Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Version 1 (updated August 2011). Cochrane Collaboration Qualitative Methods Group, 2011. Available from URL http://qim.cochrane.org/supplemental-handbook-guidance .

Centre for Reviews and Dissemination. Systematic reviews: CRD’s guidance for undertaking reviews in health care. 2009. York: University of York.

Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J. 2015;32(3):220–35.

Article   PubMed   Google Scholar  

Barroso J, Gollop C, Sandelowski M, Meynell J, Pearce P, Collins L. The challenge of searching for and retrieving qualitative studies. West J Nurs Res. 2003;25:153–78.

Ramer SL. Site-ation pearl growing: methods and librarianship history and theory. J Med Libr Assoc. 2005;93(3):397–400.

PubMed   PubMed Central   Google Scholar  

Booth A. "Brimful of STARLITE": toward standards for reporting literature searches. J Med Libr Assoc. 2006;94(4):421–9.e205.

Cooke A, Smith D, Booth A, Beyond PICO. The SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012;22:1435–43.

Evans D. Database searches for qualitative research. J Med Libr Assoc. 2002;90:290–3.

Finfgeld-Connett D, Johnson ED. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. J Adv Nurs. 2013;69(1):194–204.

Flemming K, Briggs M. Electronic searching to locate qualitative research: evaluation of three strategies. J Adv Nurs. 2007;57(1):95–100.

Grant MJ. How does your searching grow? A survey of search preferences and the use of optimal search strategies in the identification of qualitative research. Health Info Libr J. 2004;21:21–32.

Grayson L, Gomersall A. A difficult business: finding the evidence for social science reviews. ESRC UK Centre for Evidence Based Policy and Practice. Working paper 2003. London: Queen Mary, University of London.

McKibbon KA, Wilczynski NL, Haynes RB. Developing optimal search strategies for retrieving qualitative studies in PsycINFO. Eval Health Prof. 2006;29:440–54.

Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Info Libr J. 2010;27:114–22.

Shaw RL, Booth A, Sutton A, Miller T, Smith J, Young B, et al. Finding qualitative research: an evaluation of search strategies. BMC Med Res Methodol. 2004;4:5.

Subirana M, Sola I, Garcia JM, Gich I, Urrutia G. A nursing qualitative systematic review required MEDLINE and CINAHL for study identification. J Clin Epidemiol. 2005;58:20–5.

Walters LA, Wilczynski NL, Haynes RB. Developing optimal search strategies for retrieving clinically relevant qualitative studies in EMBASE. Qual Health Res. 2006;16:162–8.

Wilczynski NL, Marks S, Haynes RB. Search strategies for identifying qualitative studies in CINAHL. Qual Health Res. 2007;17(5):705–10.

Wong SSL, Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically relevant qualitative. Medinfo. 2004;107:311.

Dixon-Woods M, Fitzpatrick R. Qualitative research in systematic reviews: has established a place for itself. BMJ. 2001;323:765.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Mehdyzadeh H. Searching for the evidence: an introduction to social science information retrieval. London: Department for Culture Media and Sport; 2004.

Sandelowski M, Barroso J. Searching for and retrieving qualitative research reports. In: Sandelowski M, Barroso J, editors. Handbook for synthesizing qualitative research. New York: Springer; 2007. p. 35–74.

Schucan Bird K, Tripney J. Systematic literature searching in policy relevant, inter-disciplinary reviews: an example from culture and sport. Res Synth Methods. 2011;2(3):163–73.

Dixon-Woods M, Bonas S, Booth A, Jones DR, Miller T, Sutton AJ, et al. How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006;6:27–44.

Petticrew M, Roberts H. How to find the studies: the literature search. In: Petticrew M, Roberts H, editors. Systematic reviews in the social sciences: A practical guide. Oxford: Blackwell Publishing; 2006. p. 79–124.

Chapter   Google Scholar  

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8(1):45.

Ring N, Jepson R, Ritchie K. Methods of synthesising qualitative research for health technology assessment. Int J Technol Assess Health Care. 2011;27:384–90.

Ring NA, Ritchie K, Mandava L, Jepson R. A guide to synthesising qualitative research for researchers undertaking health technology assessments and systematic reviews. Scotland: NHS Quality Improvement Scotland (NHS QIS); 2011.

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. ESRC National Centre for Research Methods NCRM Working Paper Series. Number (01/09). London: Institute of Education; 2009.

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Research Methodol. 2009;9(1):59.

Campbell R, Pound P, Morgan M, Daker-White G, Britten N, Pill R, et al. Evaluating meta-ethnography: systematic analysis and synthesis of qualitative research. Health Technol Assess. 2011;15(43):1-164. Available from: http://www.journalslibrary.nihr.ac.uk/hta/volume-15/issue-43 . Accessed on 29 Apr 2016.

Booth A. Acknowledging a dual heritage for qualitative evidence synthesis: harnessing the qualitative research and systematic review research traditions (Doctoral dissertation, University of Sheffield). Sheffield: University of Sheffield; 2013.

Hannes K, Macaitis K. A move to more systematic and transparent approaches in qualitative evidence synthesis: update on a review of published papers. Qual Res. 2012;12(4):402–42.

Gallacher K, Jani B, Morrison D, Macdonald S, Blane D, Erwin P, et al. Qualitative systematic reviews of treatment burden in stroke, heart failure and diabetes—methodological challenges and solutions. BMC Med Res Methodol. 2013;13:10.

Lorenc T, Pearson M, Jamal F, Cooper C, Garside R. The role of systematic reviews of qualitative evidence in evaluating interventions: a case study. Res Synth Methods. 2012;3(1):1–10.

France EF, Ring N, Thomas R, Noyes J, Maxwell M, Jepson R. A methodological systematic review of what's wrong with meta-ethnography reporting. BMC Med Res Methodol. 2014;14:119.

Dixon-Woods M, Fitzpatrick R, Roberts K. Including qualitative research in systematic reviews: opportunities and problems. J Eval Clin Pract. 2000;7:125–33.

Gorecki C, Brown JM, Briggs M, Nixon J. Evaluation of five search strategies in retrieving qualitative patient-reported electronic data on the impact of pressure ulcers on quality of life. J Adv Nurs. 2010;66:645–52.

Atkins S, Lewin S, Smith H, Engel M, Fretheim A, Volmink J. Conducting a meta-ethnography of qualitative literature: lessons learnt. BMC Med Res Methodol. 2008;8:21.

Toye F, Seers K, Allcock N, Briggs M, Carr E, Barker K. Meta-ethnography 25 years on: challenges and insights for synthesising a large number of qualitative studies. BMC Med Res Methodol. 2014;14(1):80.

Booth, A, Papaioannou, D, Sutton, A J. Systematic approaches to a successful literature review. London: Sage publications Ltd; 2011.

Gough D, Oliver S, Thomas J. An introduction to systematic reviews. London: Sage Publications Ltd; 2012.

Manning N. Conclusion. In: Hannes K, Lockwood C, editors. Synthesizing qualitative research: Choosing the right approach. West Sussex: Wiley-Blackwell; 2012.

Jensen LA, Allen MN. Meta-synthesis of qualitative findings. Qual Health Res. 1996;6(4):553–60.

Sandelowski M, Docherty S, Emden C. Qualitative metasynthesis: issues and techniques. Res Nurs Health. 1997;20:365–71.

Article   CAS   PubMed   Google Scholar  

Schreiber R, Crooks D, Stern PN. Qualitative meta-analysis. In: Morse JM, editor. Completing a qualitative project: Details and dialogue. Thousand Oaks: Sage; 1997. p. 311–26.

Sherwood G. Meta-synthesis: merging qualitative studies to develop nursing knowledge. Int J Human Caring. 1999;3(1):37–42.

Booth A. Cochrane or cock-eyed? How should we conduct systematic reviews of qualitative research? Qualitative Evidence-based Practice Conference, Taking a Critical Stance; May 14 – 16; Coventry University. 2001. http://www.leeds.ac.uk/educol/documents/00001724.htm . Accessed 18 Jun 2015.

Paterson BL, Thorne S, Canam C, Jillings C. Meta-study of qualitative health research: a practical guide to meta-analysis and metasynthesis. Thousand Oaks: Sage; 2001.

Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, et al. Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Soc Sci Med. 2003;56(4):671–84.

Doyle LH. Synthesis through meta-ethnography: paradoxes, enhancements, and possibilities. Qual Res. 2003;3(3):321–44.

Finfgeld DL. Meta-synthesis: the state of the art—so far. Qual Health Res. 2003;13:893–904.

Dixon-Woods M, Agarwal S, Young B, Jones D, Sutton A. Integrative approaches to qualitative and quantitative evidence. London: Health Development Agency; 2004.

McDermott E, Graham H, Hamilton V. Experiences of being a teenage mother in the UK: a report of a systematic review of qualitative studies. Lancaster: Lancaster University; 2004.

Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist synthesis: an introduction. Manchester: ESRC Research Methods Programme Working Paper Series; 2004.

Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005;331:1064–5.

Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health. 2005;59(9):804–8.

Walsh D, Downe S. Meta-synthesis method for qualitative research: a literature review. J Adv Nurs. 2005;50:204–11.

Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, et al. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006;6(1):35.

Pawson R. Chapter 4—realist synthesis: new protocols for systematic review. In: Pawson R, editor. Evidence-based policy: a realist perspective. London: Sage Publications; 2006. p. 73–104.

Bondas T, Hall EOC. Challenges in approaching meta-synthesis research. Qual Health Res. 2007;17:113–21.

Briggs M, Flemming K. Living with leg ulceration: a synthesis of qualitative research. J Adv Nurs. 2007;59(4):319–28.

Gough D. Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. Res Pap Educ. 2007;22:2,213–228.

Weed MA. Potential method for the interpretive synthesis of qualitative research: issues in the development of meta-interpretation. Int J Social Res Methodol. 2007;11(1):13–28.

Downe S. Metasynthesis: a guide to knitting smoke. Evid Based Midwifery. 2008;6:4–8.

Corbin-Staton AP. Contexts of parental involvement: an interpretive synthesis of qualitative literature using the meta-interpretation method (Doctoral dissertation, The George Washington University). Washington DC: George Washington University; 2009.

O’Connell R, Downe S. A metasynthesis of midwives’ experience of hospital practice in publicly funded settings: compliance, resistance and authenticity. Health (London). 2009;13:589–609.

Brunton G, Wiggins M, Oakley A. Becoming a mother: a research synthesis of women’s views on the experience of first-time motherhood. ESRC End of Award Report, RES-000-22-2250. Swindon: Economic and Social Research Council; 2010.

Baxter S, Killoran A, Kelly M, Goyder E. Synthesising diverse evidence: the use of primary qualitative data analysis methods and logic models in public health reviews. Public Health. 2010;124(2):99–106.

Brunton G, Stansfield C, Thomas J. Finding relevant studies. In: Gough D, Oliver S, Thomas J, editors. An Introduction to Systematic Reviews. London: Sage Publications; 2012. p. 107–35.

Wu YP, Aylward BS, Roberts MC, Evans SC. Searching the scientific literature: implications for quantitative and qualitative reviews. Clin Psychol Rev. 2012;32:553–7.

Suri H. Purposeful sampling in qualitative research synthesis. Qual Res J. 2011;11:63–75.

Suri H. Towards methodologically inclusive research syntheses: expanding possibilities. Abingdon, Oxon: Routledge; 2013.

Lefebvre C, Manheimer E, & Glanville J. Chapter 6: searching for studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. Available from: http://handbook.cochrane.org/chapter_6/6_searching_for_studies.htm . Accessed on 29 Apr 2016.

Russell CK, Bunting SM, Gregory DM. Protective care-receiving: the active role of care recipients. J Adv Nurs. 1997;25:532–40.

Kastner M, Straus SE, McKibbon KA, Goldsmith CH. The capture-mark-recapture technique can be used as a stopping rule when searching in systematic reviews. J Clin Epidemiol. 2009;62:149–57.

Barbour RS, Barbour M. Evaluating and synthesising qualitative research: the need to develop a distinctive approach. J Eval Clin Pract. 2003;9:179–86.

Dixon-Woods M, Booth A, Sutton AJ. Synthesizing qualitative research: a review of published reports. Qual Res. 2007;7:375–422.

Coad J, Hardicre J, Devitt P. Searching for and using grey literature. Nurs Times. 2007;102:35–6.

Saini M, Shlonsky A. Systematic synthesis of qualitative research. Oxford: Oxford University Press; 2012.

McFadden P, Taylor BJ, Campbell A, McQuilkin J. Systematically identifying relevant research case study on child protection social workers’ resilience. Res Soc Work Prac. 2012;22(6):626–36.

Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. 2012;12(1):181.

McGowan J, Sampson M. Systematic reviews need systematic searchers. JMLA. 2005;93:74–80.

Franzel B, Schwiegershausen M, Heusser P, Berger B. How to locate and appraise qualitative research in complementary and alternative medicine. BMC Complement Altern Med. 2013;13(1):125.

McGinn, T, Taylor B, McColgan M, McQuilkan J. Social work literature searching current issues with databases and online search engines. Res Social Work Practice, 2014; doi 10.1177/1049731514549423 .

Stansfield C, Kavanagh J, Rees R, Gomersall A, Thomas J. The selection of search sources influences the findings of a systematic review of people's views: a case study in public health. BMC Med Res Methodol. 2012;12:55.

Stansfield C, Brunton G, Rees R. Search wide, dig deep: literature searching for qualitative research. An analysis of the publication formats and information sources used for four systematic reviews in public health. Res Synth Methods. 2014;5:142–51.

Wright K, Golder S, Lewis-Light K. What value is the CINAHL database when searching for systematic reviews of qualitative studies? Syst Rev. 2015;4:104.

Wildridge V, Bell L. How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information. Health Info Libr J. 2002;19:113–5.

Booth A. Using research in practice: Australian supermodel? A practical example of evidence-based library and information practice. Health Info Libr J. 2006;23:69–72.

Denyer D, Tranfield D. Producing a systematic review. In: Buchanan DA, Bryman A, editors. The SAGE handbook of organizational research methods. London: SAGE Publications Ltd; 2009. p. 671–89.

Howell Major C, Savin-Baden M. Designing the synthesis In: Major, C. H., Savin-Baden, M. An introduction to qualitative research synthesis: managing the information explosion in social science research. London: Routledge; 2010. p. 43–55.

Davies KS. Formulating the evidence based practice question: a review of the frameworks. Evid Based Libr Info Pract. 2011;6(2):75–80.

Joanna Briggs Institute. Joanna Briggs Institute reviewers’ manual: 2011 edition Adelaide, South Australia: University of Adelaide; 2011. Available from: http://joannabriggs.org/assets/docs/sumari/reviewersmanual-2011.pdf . (Accessed on 29 December 2015).

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Stern C, Jordan Z, McArthur A. Developing the review question and inclusion criteria. AJN: Am J Nurs. 2014;114:53–6.

Mackay G. Is there a need to differentiate between qualitative and quantitative searching strategies for literature reviewing? Qual Soc Work. 2007;6:231–41.

Glenton C, Lewin S, Scheel IB. Still too little qualitative research to shed light on results from reviews of effectiveness trials: a case study of a Cochrane review on the use of lay health workers. Implement Sci. 2011;6:53.

Lewin S, Glenton C, Oxman AD. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ. 2009;339:b3496.

Jenkins M. Evaluation of methodological search filters—a review. Health Info Libr J. 2004;21:148–63.

Bates MJ. The design of browsing and berrypicking techniques for the online search interface. Online Rev. 1989;13:407–24.

Downe S, Simpson L, Trafford K. Expert intrapartum maternity care: a metasynthesis. J Adv Nurs. 2007;57:127–40.

Pearson M, Moxham T, Ashton K. Effectiveness of search strategies for qualitative research about barriers and facilitators of program delivery. Eval Health Prof. 2011;34:297–308.

Booth A, Harris J, Croot E, Springett J, Campbell F, Wilkins E. Towards a methodology for cluster searching to provide conceptual and contextual “richness” for systematic reviews of complex interventions: case study (CLUSTER). BMC Med Res Methodol. 2013;13:118.

Beahler CC, Sundheim JJ, Trapp NI. Information retrieval in systematic reviews: challenges in the public health arena. Am J Prev Med. 2000;18:6–10.

Downe S, Finlayson K, Walsh D, Lavender T. ‘Weighing up and balancing out’: a meta-synthesis of barriers to antenatal care for marginalised women in high-income countries. Br J Obstet Gynaecol. 2009;116:518–29.

Article   CAS   Google Scholar  

Hawker S, Payne S, Kerr C, Hardey M, Powell J. Appraising the evidence: reviewing disparate data systematically. Qual Health Res. 2002;12:1284–99.

Malpass A, Carel H, Ridd M, Shaw A, Kessler D, Sharp D, et al. Transforming the perceptual situation: a meta-ethnography of qualitative work reporting patients’ experiences of mindfulness-based approaches. Mindfulness. 2012;3:60–75.

Gomersall T, Madill A, Summers LK. A metasynthesis of the self management of type 2 diabetes. Qual Health Res. 2011;21:853–71.

Steen M, Downe S, Bamford N, Edozien L. Not-patient and not-visitor: a metasynthesis fathers’ encounters with pregnancy, birth and maternity care. Midwifery. 2012;28:422–31.

Kable AK, Pich J, Maslin-Prothero SE. A structured approach to documenting a search strategy for publication: a 12 step guideline for authors. Nurse Educ Today. 2012;32:878–86.

Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: meta-narrative reviews. BMC Med. 2013;11:20.

Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: realist syntheses. BMC Med. 2013;11:21.

France EF, Ring N, Noyes J, Maxwell M, Jepson R, Duncan E, et al. Protocol-developing meta-ethnography reporting guidelines (eMERGe). BMC Med Res Methodol. 2015;15:103.

Onwuegbuzie AJ, Collins KMT. A typology of mixed methods sampling designs in social science research. Qual Rep. 2007;12:281–316.

Mays N, Pope C. Qualitative research in health care. London: BMJ Publishing Group; 1996.

Patton MQ. Qualitative evaluation and research methods. Thousand Oaks: Sage; 1990.

Baker SE, Edwards R: How many qualitative interviews is enough? National Centre for Research Methods review paper: expert voices and early career reflections on sampling and cases in qualitative research. 2012. http://eprints.ncrm.ac.uk/2273/ (last accessed 12th September 2015).

Hood WW, Wilson CS. The scatter of documents over databases in different subject domains: how many databases are needed? JASIST. 2001;52:1242–54.

Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study Health Technol Assess. 2003;7:1–76.

CAS   PubMed   Google Scholar  

Littell JH, Corcoran J, Pillai V. Systematic reviews and meta-analysis. New York: Oxford University Press; 2008.

Book   Google Scholar  

Moher D, Liberati A, Tetzlaff J, Altman D, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA Statement. PLoS Med. 2009;6, e1000097.

Higgins JPT, Green S, (editors). Cochrane handbook for systematic reviews of interventions. Version 5.0.0 [updated February 2008]: The Cochrane Collaboration; 2008. Available from: http://handbook.cochrane.org/ . Accessed on 29 Apr 2016.

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, et al. Integrating qualitative research with trials in systematic reviews. BMJ: Br Med J. 2004;328:1010.

Montori VM, Wilczynski NL, Morgan D, Haynes RB, Hedges Team. Optimal search strategies for retrieving systematic reviews from MEDLINE: analytical survey. BMJ. 2005;330(7482):68.

Cooper H. Research synthesis and meta-analysis: a step-by-step approach (4th ed., Vol. 2). London: Sage; 2010.

Niederstadt C, Droste S. Reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment. Int J Technol Assess Health Care. 2010;26:450.

Clapton J. Bibliographic databases for social care searching. London: Social Care Institute for Excellence; 2010.

Matthews EJ, Edwards AG, Barker J, Bloor M, Covey J, Hood K, et al. Efficient literature searching in diffuse topics: lessons from a systematic review of research on communicating risk to patients in primary care. Health Libr Rev. 1999;16(2):112–20.

McNally R, Alborz A. Developing methods for systematic reviewing in health services delivery and organisation: an example from a review of access to health care for people with learning disabilities. Part 1. Identifying the literature. Health Info Libr J. 2004;21(3):182–92.

Taylor BJ, Dempster M, Donnelly M. Hidden gems: systematically searching electronic databases for research publications for social work and social care. Br J Soc Work. 2003;33(4):423–39.

Taylor B, Wylie E, Dempster M, Donnelly M. Systematically retrieving research: a case study evaluating seven databases. Res Soc Work Pract. 2007;17:697–706.

Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP J Club. 1995;123:A12–3.

Ananadiou S, Procter R, Rea B, Sasaki Y, Thomas J. Supporting systematic reviews using text mining. Soc Sci Comput Rev. 2009;27:509–23.

Thomas J, McNaught J, Ananadiou S. Applications of text mining within systematic reviews. Res Synth Methods. 2011;2:1–14.

Armstrong R, Waters E, Jackson N, Oliver S, Popay J, Shepherd J, et al. Guidelines for systematic reviews of health promotion and public health interventions. Version 2. Australia: Melbourne University; 2007.

Lefebvre C, Glanville J, Wieland LS, Coles B, Weightman AL. Methodological developments in searching for studies for systematic reviews: past, present and future. Sys Rev. 2013;2:78.

Download references

Author information

Authors and affiliations.

Reader in Evidence Based Information Practice, School of Health and Related Research (ScHARR), University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK

Andrew Booth

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Andrew Booth .

Additional information

Competing interests.

The author declares that no actual or potential conflict of interest in relation to this article exists, other than authorship of multiple included articles.

The author is a volunteer co-convenor for the Cochrane Qualitative and Implementation Methods Group with specific responsibility for maintaining the Group’s study register. He is also a founder member of the Cochrane Information Retrieval Methods Group. He does not receive any remuneration in connection with information retrieval methods activities.

Authors’ contributions

AB searched and checked the published literature and guidance and extracted all data. This overview is entirely the author’s own work.

((("meta ethnography" OR "meta ethnographic") OR ("meta synthesis") OR (synthesis AND ("qualitative literature" OR "qualitative research")) OR ("critical interpretive synthesis") OR ("systematic review" AND ("qualitative research" OR "qualitative literature" OR "qualitative studies")) OR ("thematic synthesis" OR "framework synthesis") OR ("realist review" OR "realist synthesis") OR ((("qualitative systematic review" OR "qualitative evidence synthesis")) OR ("qualitative systematic reviews" OR "qualitative evidence syntheses")) OR (("quality assessment" OR "critical appraisal") AND ("qualitative research" OR "qualitative literature" OR "qualitative studies")) OR (("literature search" OR "literature searching" OR "literature searches") AND ("qualitative research" OR "qualitative literature" OR "qualitative studies")) OR (Noblit AND Hare)) OR ("meta narrative" OR "meta narratives" OR "narrative synthesis") OR ("realist reviews" OR "meta study" OR "meta method" OR "meta triangulation")) OR (CERQUAL OR CONQUAL).

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Booth, A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev 5 , 74 (2016). https://doi.org/10.1186/s13643-016-0249-x

Download citation

Received : 14 September 2015

Accepted : 21 April 2016

Published : 04 May 2016

DOI : https://doi.org/10.1186/s13643-016-0249-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative research
• Systematic reviews
• Literature searching

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• - Google Chrome

Intended for healthcare professionals

• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• News & Views
• Qualitative research...

Qualitative research in systematic reviews

• Related content
• Peer review

Has established a place for itself

• Mary Dixon-Woods , lecturer in health policy ([email protected]) ,
• Ray Fitzpatrick , professor ([email protected])
• Department of Epidemiology and Public health, University of Leicester, Leicester LE1 6TP
• Division of Public Health and Primary Health Care, Institute of Health Sciences, University of Oxford, Oxford OX3 7LF

The recent publication by the NHS Centre for Reviews and Dissemination of the second edition of the guidance on undertaking systematic reviews of research on effectiveness deserves to be warmly welcomed, for many reasons. 1 Perhaps chief among these is the increased recognition given to the diverse types of evidence that can contribute to systematic reviews. This suggests that the rigid insistence on controlled trials as the sole source of evidence on effectiveness that characterised the beginnings of the evidence based healthcare movement is fading. Qualitative research is now given explicit consideration in the new guidance. This is consistent with other recent recommendations emphasising the contribution of qualitative evidence to healthcare evaluation. 2 The argument for giving a place to qualitative research in systematic reviews seems to have been won. There remain several issues, however, that need to be addressed in making the role of qualitative evidence in reviews more systematic.

The move to recognise the potential value of qualitative research could do much to address …

Log in using your username and password

BMA Member Log In

If you have a subscription to The BMJ, log in:

• Need to activate
• Log in via institution
• Log in via OpenAthens

Log in through your institution

Subscribe from £184 *.

Subscribe and get access to all BMJ articles, and much more.

* For online subscription

Access this article for 1 day for: £50 / $60/ €56 ( excludes VAT )

You can download a PDF version for your personal record.

Buy this article

• Research article
• Open access
• Published: 10 July 2008

Methods for the thematic synthesis of qualitative research in systematic reviews

• James Thomas 1 &
• Angela Harden 1  

BMC Medical Research Methodology volume  8 , Article number:  45 ( 2008 ) Cite this article

392k Accesses

4438 Citations

105 Altmetric

Metrics details

There is a growing recognition of the value of synthesising qualitative research in the evidence base in order to facilitate effective and appropriate health care. In response to this, methods for undertaking these syntheses are currently being developed. Thematic analysis is a method that is often used to analyse data in primary qualitative research. This paper reports on the use of this type of analysis in systematic reviews to bring together and integrate the findings of multiple qualitative studies.

We describe thematic synthesis, outline several steps for its conduct and illustrate the process and outcome of this approach using a completed review of health promotion research. Thematic synthesis has three stages: the coding of text 'line-by-line'; the development of 'descriptive themes'; and the generation of 'analytical themes'. While the development of descriptive themes remains 'close' to the primary studies, the analytical themes represent a stage of interpretation whereby the reviewers 'go beyond' the primary studies and generate new interpretive constructs, explanations or hypotheses. The use of computer software can facilitate this method of synthesis; detailed guidance is given on how this can be achieved.

We used thematic synthesis to combine the studies of children's views and identified key themes to explore in the intervention studies. Most interventions were based in school and often combined learning about health benefits with 'hands-on' experience. The studies of children's views suggested that fruit and vegetables should be treated in different ways, and that messages should not focus on health warnings. Interventions that were in line with these suggestions tended to be more effective. Thematic synthesis enabled us to stay 'close' to the results of the primary studies, synthesising them in a transparent way, and facilitating the explicit production of new concepts and hypotheses.

We compare thematic synthesis to other methods for the synthesis of qualitative research, discussing issues of context and rigour. Thematic synthesis is presented as a tried and tested method that preserves an explicit and transparent link between conclusions and the text of primary studies; as such it preserves principles that have traditionally been important to systematic reviewing.

Peer Review reports

The systematic review is an important technology for the evidence-informed policy and practice movement, which aims to bring research closer to decision-making [ 1 , 2 ]. This type of review uses rigorous and explicit methods to bring together the results of primary research in order to provide reliable answers to particular questions [ 3 – 6 ]. The picture that is presented aims to be distorted neither by biases in the review process nor by biases in the primary research which the review contains [ 7 – 10 ]. Systematic review methods are well-developed for certain types of research, such as randomised controlled trials (RCTs). Methods for reviewing qualitative research in a systematic way are still emerging, and there is much ongoing development and debate [ 11 – 14 ].

In this paper we present one approach to the synthesis of findings of qualitative research, which we have called 'thematic synthesis'. We have developed and applied these methods within several systematic reviews that address questions about people's perspectives and experiences [ 15 – 18 ]. The context for this methodological development is a programme of work in health promotion and public health (HP & PH), mostly funded by the English Department of Health, at the EPPI-Centre, in the Social Science Research Unit at the Institute of Education, University of London in the UK. Early systematic reviews at the EPPI-Centre addressed the question 'what works?' and contained research testing the effects of interventions. However, policy makers and other review users also posed questions about intervention need, appropriateness and acceptability, and factors influencing intervention implementation. To address these questions, our reviews began to include a wider range of research, including research often described as 'qualitative'. We began to focus, in particular, on research that aimed to understand the health issue in question from the experiences and point of view of the groups of people targeted by HP&PH interventions (We use the term 'qualitative' research cautiously because it encompasses a multitude of research methods at the same time as an assumed range of epistemological positions. In practice it is often difficult to classify research as being either 'qualitative' or 'quantitative' as much research contains aspects of both [ 19 – 22 ]. Because the term is in common use, however, we will employ it in this paper).

When we started the work for our first series of reviews which included qualitative research in 1999 [ 23 – 26 ], there was very little published material that described methods for synthesising this type of research. We therefore experimented with a variety of techniques borrowed from standard systematic review methods and methods for analysing primary qualitative research [ 15 ]. In later reviews, we were able to refine these methods and began to apply thematic analysis in a more explicit way. The methods for thematic synthesis described in this paper have so far been used explicitly in three systematic reviews [ 16 – 18 ].

The review used as an example in this paper

To illustrate the steps involved in a thematic synthesis we draw on a review of the barriers to, and facilitators of, healthy eating amongst children aged four to 10 years old [ 17 ]. The review was commissioned by the Department of Health, England to inform policy about how to encourage children to eat healthily in the light of recent surveys highlighting that British children are eating less than half the recommended five portions of fruit and vegetables per day. While we focus on the aspects of the review that relate to qualitative studies, the review was broader than this and combined answering traditional questions of effectiveness, through reviewing controlled trials, with questions relating to children's views of healthy eating, which were answered using qualitative studies. The qualitative studies were synthesised using 'thematic synthesis' – the subject of this paper. We compared the effectiveness of interventions which appeared to be in line with recommendations from the thematic synthesis with those that did not. This enabled us to see whether the understandings we had gained from the children's views helped us to explain differences in the effectiveness of different interventions: the thematic synthesis had enabled us to generate hypotheses which could be tested against the findings of the quantitative studies – hypotheses that we could not have generated without the thematic synthesis. The methods of this part of the review are published in Thomas et al . [ 27 ] and are discussed further in Harden and Thomas [ 21 ].

Qualitative research and systematic reviews

The act of seeking to synthesise qualitative research means stepping into more complex and contested territory than is the case when only RCTs are included in a review. First, methods are much less developed in this area, with fewer completed reviews available from which to learn, and second, the whole enterprise of synthesising qualitative research is itself hotly debated. Qualitative research, it is often proposed, is not generalisable and is specific to a particular context, time and group of participants. Thus, in bringing such research together, reviewers are open to the charge that they de-contextualise findings and wrongly assume that these are commensurable [ 11 , 13 ]. These are serious concerns which it is not the purpose of this paper to contest. We note, however, that a strong case has been made for qualitative research to be valued for the potential it has to inform policy and practice [ 11 , 28 – 30 ]. In our experience, users of reviews are interested in the answers that only qualitative research can provide, but are not able to handle the deluge of data that would result if they tried to locate, read and interpret all the relevant research themselves. Thus, if we acknowledge the unique importance of qualitative research, we need also to recognise that methods are required to bring its findings together for a wide audience – at the same time as preserving and respecting its essential context and complexity.

The earliest published work that we know of that deals with methods for synthesising qualitative research was written in 1988 by Noblit and Hare [ 31 ]. This book describes the way that ethnographic research might be synthesised, but the method has been shown to be applicable to qualitative research beyond ethnography [ 32 , 11 ]. As well as meta-ethnography, other methods have been developed more recently, including 'meta-study' [ 33 ], 'critical interpretive synthesis' [ 34 ] and 'metasynthesis' [ 13 ].

Many of the newer methods being developed have much in common with meta-ethnography, as originally described by Noblit and Hare, and often state explicitly that they are drawing on this work. In essence, this method involves identifying key concepts from studies and translating them into one another. The term 'translating' in this context refers to the process of taking concepts from one study and recognising the same concepts in another study, though they may not be expressed using identical words. Explanations or theories associated with these concepts are also extracted and a 'line of argument' may be developed, pulling corroborating concepts together and, crucially, going beyond the content of the original studies (though 'refutational' concepts might not be amenable to this process). Some have claimed that this notion of 'going beyond' the primary studies is a critical component of synthesis, and is what distinguishes it from the types of summaries of findings that typify traditional literature reviews [e.g. [ 32 ], p209]. In the words of Margarete Sandelowski, "metasyntheses are integrations that are more than the sum of parts, in that they offer novel interpretations of findings. These interpretations will not be found in any one research report but, rather, are inferences derived from taking all of the reports in a sample as a whole" [[ 14 ], p1358].

Thematic analysis has been identified as one of a range of potential methods for research synthesis alongside meta-ethnography and 'metasynthesis', though precisely what the method involves is unclear, and there are few examples of it being used for synthesising research [ 35 ]. We have adopted the term 'thematic synthesis', as we translated methods for the analysis of primary research – often termed 'thematic' – for use in systematic reviews [ 36 – 38 ]. As Boyatzis [[ 36 ], p4] has observed, thematic analysis is "not another qualitative method but a process that can be used with most, if not all, qualitative methods..." . Our approach concurs with this conceptualisation of thematic analysis, since the method we employed draws on other established methods but uses techniques commonly described as 'thematic analysis' in order to formalise the identification and development of themes.

We now move to a description of the methods we used in our example systematic review. While this paper has the traditional structure for reporting the results of a research project, the detailed methods (e.g. precise terms we used for searching) and results are available online. This paper identifies the particular issues that relate especially to reviewing qualitative research systematically and then to describing the activity of thematic synthesis in detail.

When searching for studies for inclusion in a 'traditional' statistical meta-analysis, the aim of searching is to locate all relevant studies. Failing to do this can undermine the statistical models that underpin the analysis and bias the results. However, Doyle [[ 39 ], p326] states that, "like meta-analysis, meta-ethnography utilizes multiple empirical studies but, unlike meta-analysis, the sample is purposive rather than exhaustive because the purpose is interpretive explanation and not prediction" . This suggests that it may not be necessary to locate every available study because, for example, the results of a conceptual synthesis will not change if ten rather than five studies contain the same concept, but will depend on the range of concepts found in the studies, their context, and whether they are in agreement or not. Thus, principles such as aiming for 'conceptual saturation' might be more appropriate when planning a search strategy for qualitative research, although it is not yet clear how these principles can be applied in practice. Similarly, other principles from primary qualitative research methods may also be 'borrowed' such as deliberately seeking studies which might act as negative cases, aiming for maximum variability and, in essence, designing the resulting set of studies to be heterogeneous, in some ways, instead of achieving the homogeneity that is often the aim in statistical meta-analyses.

However you look, qualitative research is difficult to find [ 40 – 42 ]. In our review, it was not possible to rely on simple electronic searches of databases. We needed to search extensively in 'grey' literature, ask authors of relevant papers if they knew of more studies, and look especially for book chapters, and we spent a lot of effort screening titles and abstracts by hand and looking through journals manually. In this sense, while we were not driven by the statistical imperative of locating every relevant study, when it actually came down to searching, we found that there was very little difference in the methods we had to use to find qualitative studies compared to the methods we use when searching for studies for inclusion in a meta-analysis.

Quality assessment

Assessing the quality of qualitative research has attracted much debate and there is little consensus regarding how quality should be assessed, who should assess quality, and, indeed, whether quality can or should be assessed in relation to 'qualitative' research at all [ 43 , 22 , 44 , 45 ]. We take the view that the quality of qualitative research should be assessed to avoid drawing unreliable conclusions. However, since there is little empirical evidence on which to base decisions for excluding studies based on quality assessment, we took the approach in this review to use 'sensitivity analyses' (described below) to assess the possible impact of study quality on the review's findings.

In our example review we assessed our studies according to 12 criteria, which were derived from existing sets of criteria proposed for assessing the quality of qualitative research [ 46 – 49 ], principles of good practice for conducting social research with children [ 50 ], and whether studies employed appropriate methods for addressing our review questions. The 12 criteria covered three main quality issues. Five related to the quality of the reporting of a study's aims, context, rationale, methods and findings (e.g. was there an adequate description of the sample used and the methods for how the sample was selected and recruited?). A further four criteria related to the sufficiency of the strategies employed to establish the reliability and validity of data collection tools and methods of analysis, and hence the validity of the findings. The final three criteria related to the assessment of the appropriateness of the study methods for ensuring that findings about the barriers to, and facilitators of, healthy eating were rooted in children's own perspectives (e.g. were data collection methods appropriate for helping children to express their views?).

Extracting data from studies

One issue which is difficult to deal with when synthesising 'qualitative' studies is 'what counts as data' or 'findings'? This problem is easily addressed when a statistical meta-analysis is being conducted: the numeric results of RCTs – for example, the mean difference in outcome between the intervention and control – are taken from published reports and are entered into the software package being used to calculate the pooled effect size [ 3 , 51 ].

Deciding what to abstract from the published report of a 'qualitative' study is much more difficult. Campbell et al . [ 11 ] extracted what they called the 'key concepts' from the qualitative studies they found about patients' experiences of diabetes and diabetes care. However, finding the key concepts in 'qualitative' research is not always straightforward either. As Sandelowski and Barroso [ 52 ] discovered, identifying the findings in qualitative research can be complicated by varied reporting styles or the misrepresentation of data as findings (as for example when data are used to 'let participants speak for themselves'). Sandelowski and Barroso [ 53 ] have argued that the findings of qualitative (and, indeed, all empirical) research are distinct from the data upon which they are based, the methods used to derive them, externally sourced data, and researchers' conclusions and implications.

In our example review, while it was relatively easy to identify 'data' in the studies – usually in the form of quotations from the children themselves – it was often difficult to identify key concepts or succinct summaries of findings, especially for studies that had undertaken relatively simple analyses and had not gone much further than describing and summarising what the children had said. To resolve this problem we took study findings to be all of the text labelled as 'results' or 'findings' in study reports – though we also found 'findings' in the abstracts which were not always reported in the same way in the text. Study reports ranged in size from a few pages to full final project reports. We entered all the results of the studies verbatim into QSR's NVivo software for qualitative data analysis. Where we had the documents in electronic form this process was straightforward even for large amounts of text. When electronic versions were not available, the results sections were either re-typed or scanned in using a flat-bed or pen scanner. (We have since adapted our own reviewing system, 'EPPI-Reviewer' [ 54 ], to handle this type of synthesis and the screenshots below show this software.)

Detailed methods for thematic synthesis

The synthesis took the form of three stages which overlapped to some degree: the free line-by-line coding of the findings of primary studies; the organisation of these 'free codes' into related areas to construct 'descriptive' themes; and the development of 'analytical' themes.

Stages one and two: coding text and developing descriptive themes

In our children and healthy eating review, we originally planned to extract and synthesise study findings according to our review questions regarding the barriers to, and facilitators of, healthy eating amongst children. It soon became apparent, however, that few study findings addressed these questions directly and it appeared that we were in danger of ending up with an empty synthesis. We were also concerned about imposing the a priori framework implied by our review questions onto study findings without allowing for the possibility that a different or modified framework may be a better fit. We therefore temporarily put our review questions to one side and started from the study findings themselves to conduct an thematic analysis.

There were eight relevant qualitative studies examining children's views of healthy eating. We entered the verbatim findings of these studies into our database. Three reviewers then independently coded each line of text according to its meaning and content. Figure 1 illustrates this line-by-line coding using our specialist reviewing software, EPPI-Reviewer, which includes a component designed to support thematic synthesis. The text which was taken from the report of the primary study is on the left and codes were created inductively to capture the meaning and content of each sentence. Codes could be structured, either in a tree form (as shown in the figure) or as 'free' codes – without a hierarchical structure.

line-by-line coding in EPPI-Reviewer.

The use of line-by-line coding enabled us to undertake what has been described as one of the key tasks in the synthesis of qualitative research: the translation of concepts from one study to another [ 32 , 55 ]. However, this process may not be regarded as a simple one of translation. As we coded each new study we added to our 'bank' of codes and developed new ones when necessary. As well as translating concepts between studies, we had already begun the process of synthesis (For another account of this process, see Doyle [[ 39 ], p331]). Every sentence had at least one code applied, and most were categorised using several codes (e.g. 'children prefer fruit to vegetables' or 'why eat healthily?'). Before completing this stage of the synthesis, we also examined all the text which had a given code applied to check consistency of interpretation and to see whether additional levels of coding were needed. (In grounded theory this is termed 'axial' coding; see Fisher [ 55 ] for further discussion of the application of axial coding in research synthesis.) This process created a total of 36 initial codes. For example, some of the text we coded as "bad food = nice, good food = awful" from one study [ 56 ] were:

'All the things that are bad for you are nice and all the things that are good for you are awful.' (Boys, year 6) [[ 56 ], p74]

'All adverts for healthy stuff go on about healthy things. The adverts for unhealthy things tell you how nice they taste.' [[ 56 ], p75]

Some children reported throwing away foods they knew had been put in because they were 'good for you' and only ate the crisps and chocolate . [[ 56 ], p75]

Reviewers looked for similarities and differences between the codes in order to start grouping them into a hierarchical tree structure. New codes were created to capture the meaning of groups of initial codes. This process resulted in a tree structure with several layers to organize a total of 12 descriptive themes (Figure 2 ). For example, the first layer divided the 12 themes into whether they were concerned with children's understandings of healthy eating or influences on children's food choice. The above example, about children's preferences for food, was placed in both areas, since the findings related both to children's reactions to the foods they were given, and to how they behaved when given the choice over what foods they might eat. A draft summary of the findings across the studies organized by the 12 descriptive themes was then written by one of the review authors. Two other review authors commented on this draft and a final version was agreed.

relationships between descriptive themes.

Stage three: generating analytical themes

Up until this point, we had produced a synthesis which kept very close to the original findings of the included studies. The findings of each study had been combined into a whole via a listing of themes which described children's perspectives on healthy eating. However, we did not yet have a synthesis product that addressed directly the concerns of our review – regarding how to promote healthy eating, in particular fruit and vegetable intake, amongst children. Neither had we 'gone beyond' the findings of the primary studies and generated additional concepts, understandings or hypotheses. As noted earlier, the idea or step of 'going beyond' the content of the original studies has been identified by some as the defining characteristic of synthesis [ 32 , 14 ].

This stage of a qualitative synthesis is the most difficult to describe and is, potentially, the most controversial, since it is dependent on the judgement and insights of the reviewers. The equivalent stage in meta-ethnography is the development of 'third order interpretations' which go beyond the content of original studies [ 32 , 11 ]. In our example, the step of 'going beyond' the content of the original studies was achieved by using the descriptive themes that emerged from our inductive analysis of study findings to answer the review questions we had temporarily put to one side. Reviewers inferred barriers and facilitators from the views children were expressing about healthy eating or food in general, captured by the descriptive themes, and then considered the implications of children's views for intervention development. Each reviewer first did this independently and then as a group. Through this discussion more abstract or analytical themes began to emerge. The barriers and facilitators and implications for intervention development were examined again in light of these themes and changes made as necessary. This cyclical process was repeated until the new themes were sufficiently abstract to describe and/or explain all of our initial descriptive themes, our inferred barriers and facilitators and implications for intervention development.

For example, five of the 12 descriptive themes concerned the influences on children's choice of foods (food preferences, perceptions of health benefits, knowledge behaviour gap, roles and responsibilities, non-influencing factors). From these, reviewers inferred several barriers and implications for intervention development. Children identified readily that taste was the major concern for them when selecting food and that health was either a secondary factor or, in some cases, a reason for rejecting food. Children also felt that buying healthy food was not a legitimate use of their pocket money, which they would use to buy sweets that could be enjoyed with friends. These perspectives indicated to us that branding fruit and vegetables as a 'tasty' rather than 'healthy' might be more effective in increasing consumption. As one child noted astutely, 'All adverts for healthy stuff go on about healthy things. The adverts for unhealthy things tell you how nice they taste.' [[ 56 ], p75]. We captured this line of argument in the analytical theme entitled 'Children do not see it as their role to be interested in health'. Altogether, this process resulted in the generation of six analytical themes which were associated with ten recommendations for interventions.

Six main issues emerged from the studies of children's views: (1) children do not see it as their role to be interested in health; (2) children do not see messages about future health as personally relevant or credible; (3) fruit, vegetables and confectionery have very different meanings for children; (4) children actively seek ways to exercise their own choices with regard to food; (5) children value eating as a social occasion; and (6) children see the contradiction between what is promoted in theory and what adults provide in practice. The review found that most interventions were based in school (though frequently with parental involvement) and often combined learning about the health benefits of fruit and vegetables with 'hands-on' experience in the form of food preparation and taste-testing. Interventions targeted at people with particular risk factors worked better than others, and multi-component interventions that combined the promotion of physical activity with healthy eating did not work as well as those that only concentrated on healthy eating. The studies of children's views suggested that fruit and vegetables should be treated in different ways in interventions, and that messages should not focus on health warnings. Interventions that were in line with these suggestions tended to be more effective than those which were not.

Context and rigour in thematic synthesis

The process of translation, through the development of descriptive and analytical themes, can be carried out in a rigorous way that facilitates transparency of reporting. Since we aim to produce a synthesis that both generates 'abstract and formal theories' that are nevertheless 'empirically faithful to the cases from which they were developed' [[ 53 ], p1371], we see the explicit recording of the development of themes as being central to the method. The use of software as described can facilitate this by allowing reviewers to examine the contribution made to their findings by individual studies, groups of studies, or sub-populations within studies.

Some may argue against the synthesis of qualitative research on the grounds that the findings of individual studies are de-contextualised and that concepts identified in one setting are not applicable to others [ 32 ]. However, the act of synthesis could be viewed as similar to the role of a research user when reading a piece of qualitative research and deciding how useful it is to their own situation. In the case of synthesis, reviewers translate themes and concepts from one situation to another and can always be checking that each transfer is valid and whether there are any reasons that understandings gained in one context might not be transferred to another. We attempted to preserve context by providing structured summaries of each study detailing aims, methods and methodological quality, and setting and sample. This meant that readers of our review were able to judge for themselves whether or not the contexts of the studies the review contained were similar to their own. In the synthesis we also checked whether the emerging findings really were transferable across different study contexts. For example, we tried throughout the synthesis to distinguish between participants (e.g. boys and girls) where the primary research had made an appropriate distinction. We then looked to see whether some of our synthesis findings could be attributed to a particular group of children or setting. In the event, we did not find any themes that belonged to a specific group, but another outcome of this process was a realisation that the contextual information given in the reports of studies was very restricted indeed. It was therefore difficult to make the best use of context in our synthesis.

In checking that we were not translating concepts into situations where they did not belong, we were following a principle that others have followed when using synthesis methods to build grounded formal theory: that of grounding a text in the context in which it was constructed. As Margaret Kearney has noted "the conditions under which data were collected, analysis was done, findings were found, and products were written for each contributing report should be taken into consideration in developing a more generalized and abstract model" [[ 14 ], p1353]. Britten et al . [ 32 ] suggest that it may be important to make a deliberate attempt to include studies conducted across diverse settings to achieve the higher level of abstraction that is aimed for in a meta-ethnography.

Study quality and sensitivity analyses

We assessed the 'quality' of our studies with regard to the degree to which they represented the views of their participants. In doing this, we were locating the concept of 'quality' within the context of the purpose of our review – children's views – and not necessarily the context of the primary studies themselves. Our 'hierarchy of evidence', therefore, did not prioritise the research design of studies but emphasised the ability of the studies to answer our review question. A traditional systematic review of controlled trials would contain a quality assessment stage, the purpose of which is to exclude studies that do not provide a reliable answer to the review question. However, given that there were no accepted – or empirically tested – methods for excluding qualitative studies from syntheses on the basis of their quality [ 57 , 12 , 58 ], we included all studies regardless of their quality.

Nevertheless, our studies did differ according to the quality criteria they were assessed against and it was important that we considered this in some way. In systematic reviews of trials, 'sensitivity analyses' – analyses which test the effect on the synthesis of including and excluding findings from studies of differing quality – are often carried out. Dixon-Woods et al . [ 12 ] suggest that assessing the feasibility and worth of conducting sensitivity analyses within syntheses of qualitative research should be an important focus of synthesis methods work. After our thematic synthesis was complete, we examined the relative contributions of studies to our final analytic themes and recommendations for interventions. We found that the poorer quality studies contributed comparatively little to the synthesis and did not contain many unique themes; the better studies, on the other hand, appeared to have more developed analyses and contributed most to the synthesis.

This paper has discussed the rationale for reviewing and synthesising qualitative research in a systematic way and has outlined one specific approach for doing this: thematic synthesis. While it is not the only method which might be used – and we have discussed some of the other options available – we present it here as a tested technique that has worked in the systematic reviews in which it has been employed.

We have observed that one of the key tasks in the synthesis of qualitative research is the translation of concepts between studies. While the activity of translating concepts is usually undertaken in the few syntheses of qualitative research that exist, there are few examples that specify the detail of how this translation is actually carried out. The example above shows how we achieved the translation of concepts across studies through the use of line-by-line coding, the organisation of these codes into descriptive themes, and the generation of analytical themes through the application of a higher level theoretical framework. This paper therefore also demonstrates how the methods and process of a thematic synthesis can be written up in a transparent way.

This paper goes some way to addressing concerns regarding the use of thematic analysis in research synthesis raised by Dixon-Woods and colleagues who argue that the approach can lack transparency due to a failure to distinguish between 'data-driven' or 'theory-driven' approaches. Moreover they suggest that, "if thematic analysis is limited to summarising themes reported in primary studies, it offers little by way of theoretical structure within which to develop higher order thematic categories..." [[ 35 ], p47]. Part of the problem, they observe, is that the precise methods of thematic synthesis are unclear. Our approach contains a clear separation between the 'data-driven' descriptive themes and the 'theory-driven' analytical themes and demonstrates how the review questions provided a theoretical structure within which it became possible to develop higher order thematic categories.

The theme of 'going beyond' the content of the primary studies was discussed earlier. Citing Strike and Posner [ 59 ], Campbell et al . [[ 11 ], p672] also suggest that synthesis "involves some degree of conceptual innovation, or employment of concepts not found in the characterisation of the parts and a means of creating the whole" . This was certainly true of the example given in this paper. We used a series of questions, derived from the main topic of our review, to focus an examination of our descriptive themes and we do not find our recommendations for interventions contained in the findings of the primary studies: these were new propositions generated by the reviewers in the light of the synthesis. The method also demonstrates that it is possible to synthesise without conceptual innovation. The initial synthesis, involving the translation of concepts between studies, was necessary in order for conceptual innovation to begin. One could argue that the conceptual innovation, in this case, was only necessary because the primary studies did not address our review question directly. In situations in which the primary studies are concerned directly with the review question, it may not be necessary to go beyond the contents of the original studies in order to produce a satisfactory synthesis (see, for example, Marston and King, [ 60 ]). Conceptually, our analytical themes are similar to the ultimate product of meta-ethnographies: third order interpretations [ 11 ], since both are explicit mechanisms for going beyond the content of the primary studies and presenting this in a transparent way. The main difference between them lies in their purposes. Third order interpretations bring together the implications of translating studies into one another in their own terms, whereas analytical themes are the result of interrogating a descriptive synthesis by placing it within an external theoretical framework (our review question and sub-questions). It may be, therefore, that analytical themes are more appropriate when a specific review question is being addressed (as often occurs when informing policy and practice), and third order interpretations should be used when a body of literature is being explored in and of itself, with broader, or emergent, review questions.

This paper is a contribution to the current developmental work taking place in understanding how best to bring together the findings of qualitative research to inform policy and practice. It is by no means the only method on offer but, by drawing on methods and principles from qualitative primary research, it benefits from the years of methodological development that underpins the research it seeks to synthesise.

Chalmers I: Trying to do more good than harm in policy and practice: the role of rigorous, transparent and up-to-date evaluations. Ann Am Acad Pol Soc Sci. 2003, 589: 22-40. 10.1177/0002716203254762.

Article   Google Scholar  

Oakley A: Social science and evidence-based everything: the case of education. Educ Rev. 2002, 54: 277-286. 10.1080/0013191022000016329.

Cooper H, Hedges L: The Handbook of Research Synthesis. 1994, New York: Russell Sage Foundation

Google Scholar  

EPPI-Centre: EPPI-Centre Methods for Conducting Systematic Reviews. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=89 ]

Higgins J, Green S, (Eds): Cochrane Handbook for Systematic Reviews of Interventions 4.2.6. 2006, Updated September 2006. Accessed 24th January 2007, [ http://www.cochrane.org/resources/handbook/ ]

Petticrew M, Roberts H: Systematic Reviews in the Social Sciences: A practical guide. 2006, Oxford: Blackwell Publishing

Book   Google Scholar  

Chalmers I, Hedges L, Cooper H: A brief history of research synthesis. Eval Health Prof. 2002, 25: 12-37. 10.1177/0163278702025001003.

Article   PubMed   Google Scholar  

Juni P, Altman D, Egger M: Assessing the quality of controlled clinical trials. BMJ. 2001, 323: 42-46. 10.1136/bmj.323.7303.42.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Mulrow C: Systematic reviews: rationale for systematic reviews. BMJ. 1994, 309: 597-599.

White H: Scientific communication and literature retrieval. The Handbook of Research Synthesis. Edited by: Cooper H, Hedges L. 1994, New York: Russell Sage Foundation

Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, Donovan J: Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Soc Sci Med. 2003, 56: 671-684. 10.1016/S0277-9536(02)00064-3.

Dixon-Woods M, Bonas S, Booth A, Jones DR, Miller T, Sutton AJ, Shaw RL, Smith JA, Young B: How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006, 6: 27-44. 10.1177/1468794106058867.

Sandelowski M, Barroso J: Handbook for Synthesising Qualitative Research. 2007, New York: Springer

Thorne S, Jensen L, Kearney MH, Noblit G, Sandelowski M: Qualitative meta-synthesis: reflections on methodological orientation and ideological agenda. Qual Health Res. 2004, 14: 1342-1365. 10.1177/1049732304269888.

Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A: Applying systematic review methods to studies of people's views: an example from public health. J Epidemiol Community Health. 2004, 58: 794-800. 10.1136/jech.2003.014829.

Article   PubMed   PubMed Central   Google Scholar  

Harden A, Brunton G, Fletcher A, Oakley A: Young People, Pregnancy and Social Exclusion: A systematic synthesis of research evidence to identify effective, appropriate and promising approaches for prevention and support. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=674 ]

Thomas J, Sutcliffe K, Harden A, Oakley A, Oliver S, Rees R, Brunton G, Kavanagh J: Children and Healthy Eating: A systematic review of barriers and facilitators. 2003, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, accessed 4 th July 2008, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=246 ]

Thomas J, Kavanagh J, Tucker H, Burchett H, Tripney J, Oakley A: Accidental Injury, Risk-Taking Behaviour and the Social Circumstances in which Young People Live: A systematic review. 2007, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=1910 ]

Bryman A: Quantity and Quality in Social Research. 1998, London: Unwin

Hammersley M: What's Wrong with Ethnography?. 1992, London: Routledge

Harden A, Thomas J: Methodological issues in combining diverse study types in systematic reviews. Int J Soc Res Meth. 2005, 8: 257-271. 10.1080/13645570500155078.

Oakley A: Experiments in Knowing: Gender and methods in the social sciences. 2000, Cambridge: Polity Press

Harden A, Oakley A, Oliver S: Peer-delivered health promotion for young people: a systematic review of different study designs. Health Educ J. 2001, 60: 339-353. 10.1177/001789690106000406.

Harden A, Rees R, Shepherd J, Brunton G, Oliver S, Oakley A: Young People and Mental Health: A systematic review of barriers and facilitators. 2001, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=256 ]

Rees R, Harden A, Shepherd J, Brunton G, Oliver S, Oakley A: Young People and Physical Activity: A systematic review of barriers and facilitators. 2001, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=260 ]

Shepherd J, Harden A, Rees R, Brunton G, Oliver S, Oakley A: Young People and Healthy Eating: A systematic review of barriers and facilitators. 2001, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=258 ]

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J: Integrating qualitative research with trials in systematic reviews: an example from public health. BMJ. 2004, 328: 1010-1012. 10.1136/bmj.328.7446.1010.

Davies P: What is evidence-based education?. Br J Educ Stud. 1999, 47: 108-121. 10.1111/1467-8527.00106.

Newman M, Thompson C, Roberts AP: Helping practitioners understand the contribution of qualitative research to evidence-based practice. Evid Based Nurs. 2006, 9: 4-7. 10.1136/ebn.9.1.4.

Popay J: Moving Beyond Effectiveness in Evidence Synthesis. 2006, London: National Institute for Health and Clinical Excellence

Noblit GW, Hare RD: Meta-Ethnography: Synthesizing qualitative studies. 1988, Newbury Park: Sage

Britten N, Campbell R, Pope C, Donovan J, Morgan M, Pill R: Using meta-ethnography to synthesise qualitative research: a worked example. J Health Serv Res Policy. 2002, 7: 209-215. 10.1258/135581902320432732.

Paterson B, Thorne S, Canam C, Jillings C: Meta-Study of Qualitative Health Research. 2001, Thousand Oaks, California: Sage

Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ: Conducting a critical interpretative synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006, 6: 35-10.1186/1471-2288-6-35.

Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A: Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005, 10: 45-53. 10.1258/1355819052801804.

Boyatzis RE: Transforming Qualitative Information. 1998, Sage: Cleveland

Braun V, Clarke V: Using thematic analysis in psychology. Qual Res Psychol. 2006, 3: 77-101. 10.1191/1478088706qp063oa. [ http://science.uwe.ac.uk/psychology/drvictoriaclarke_files/thematicanalysis%20.pdf ]

Silverman D, Ed: Qualitative Research: Theory, method and practice. 1997, London: Sage

Doyle LH: Synthesis through meta-ethnography: paradoxes, enhancements, and possibilities. Qual Res. 2003, 3: 321-344. 10.1177/1468794103033003.

Barroso J, Gollop C, Sandelowski M, Meynell J, Pearce PF, Collins LJ: The challenges of searching for and retrieving qualitative studies. Western J Nurs Res. 2003, 25: 153-178. 10.1177/0193945902250034.

Walters LA, Wilczynski NL, Haynes RB, Hedges Team: Developing optimal search strategies for retrieving clinically relevant qualitative studies in EMBASE. Qual Health Res. 2006, 16: 162-8. 10.1177/1049732305284027.

Wong SSL, Wilczynski NL, Haynes RB: Developing optimal search strategies for detecting clinically relevant qualitative studies in Medline. Medinfo. 2004, 11: 311-314.

Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P: Qualitative research methods in health technology assessment: a review of the literature. Health Technol Assess. 1998, 2 (16):

Seale C: Quality in qualitative research. Qual Inq. 1999, 5: 465-478.

Spencer L, Ritchie J, Lewis J, Dillon L: Quality in Qualitative Evaluation: A framework for assessing research evidence. 2003, London: Cabinet Office

Boulton M, Fitzpatrick R, Swinburn C: Qualitative research in healthcare II: a structured review and evaluation of studies. J Eval Clin Pract. 1996, 2: 171-179. 10.1111/j.1365-2753.1996.tb00041.x.

Article   CAS   PubMed   Google Scholar  

Cobb A, Hagemaster J: Ten criteria for evaluating qualitative research proposals. J Nurs Educ. 1987, 26: 138-143.

CAS   PubMed   Google Scholar  

Mays N, Pope C: Rigour and qualitative research. BMJ. 1995, 311: 109-12.

Medical Sociology Group: Criteria for the evaluation of qualitative research papers. Med Sociol News. 1996, 22: 68-71.

Alderson P: Listening to Children. 1995, London: Barnardo's

Egger M, Davey-Smith G, Altman D: Systematic Reviews in Health Care: Meta-analysis in context. 2001, London: BMJ Publishing

Sandelowski M, Barroso J: Finding the findings in qualitative studies. J Nurs Scholarsh. 2002, 34: 213-219. 10.1111/j.1547-5069.2002.00213.x.

Sandelowski M: Using qualitative research. Qual Health Res. 2004, 14: 1366-1386. 10.1177/1049732304269672.

Thomas J, Brunton J: EPPI-Reviewer 3.0: Analysis and management of data for research synthesis. EPPI-Centre software. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education

Fisher M, Qureshi H, Hardyman W, Homewood J: Using Qualitative Research in Systematic Reviews: Older people's views of hospital discharge. 2006, London: Social Care Institute for Excellence

Dixey R, Sahota P, Atwal S, Turner A: Children talking about healthy eating: data from focus groups with 300 9–11-year-olds. Nutr Bull. 2001, 26: 71-79. 10.1046/j.1467-3010.2001.00078.x.

Daly A, Willis K, Small R, Green J, Welch N, Kealy M, Hughes E: Hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol. 2007, 60: 43-49. 10.1016/j.jclinepi.2006.03.014.

Popay J: Moving beyond floccinaucinihilipilification: enhancing the utility of systematic reviews. J Clin Epidemiol. 2005, 58: 1079-80. 10.1016/j.jclinepi.2005.08.004.

Strike K, Posner G: Types of synthesis and their criteria. Knowledge Structure and Use: Implications for synthesis and interpretation. Edited by: Ward S, Reed L. 1983, Philadelphia: Temple University Press

Marston C, King E: Factors that shape young people's sexual behaviour: a systematic review. The Lancet. 2006, 368: 1581-86. 10.1016/S0140-6736(06)69662-1.

Pre-publication history

The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2288/8/45/prepub

Download references

Acknowledgements

The authors would like to thank Elaine Barnett-Page for her assistance in producing the draft paper, and David Gough, Ann Oakley and Sandy Oliver for their helpful comments. The review used an example in this paper was funded by the Department of Health (England). The methodological development was supported by Department of Health (England) and the ESRC through the Methods for Research Synthesis Node of the National Centre for Research Methods. In addition, Angela Harden held a senior research fellowship funded by the Department of Health (England) December 2003 – November 2007. The views expressed in this paper are those of the authors and are not necessarily those of the funding bodies.

Author information

Authors and affiliations.

EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, UK

James Thomas & Angela Harden

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to James Thomas .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors' contributions

Both authors contributed equally to the paper and read and approved the final manuscript.

James Thomas and Angela Harden contributed equally to this work.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, rights and permissions.

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Thomas, J., Harden, A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol 8 , 45 (2008). https://doi.org/10.1186/1471-2288-8-45

Download citation

Received : 17 April 2008

Accepted : 10 July 2008

Published : 10 July 2008

DOI : https://doi.org/10.1186/1471-2288-8-45

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative Research
• Primary Study
• Analytical Theme
• Healthy Eating
• Review Question

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

• Quantitative vs. Qualitative Research

Research can be   quantitative or qualitative  or both:

• A quantitative systematic review will include studies that have numerical data.
• A qualitative systematic review derives data from observation, interviews, or verbal interactions and focuses on the meanings and interpretations of the participants. It may include focus groups, interviews, observations and diaries.

Video source:  UniversityNow: Quantitative vs. Qualitative Research

For more information on searching for qualitative evidence see:

Booth, A. (2016). Searching for qualitative research for inclusion in systematic reviews: A structured methodological review.  Systematic Reviews, 5 (1), 1–23. https://doi.org/10.1186/S13643-016-0249-X/TABLES/5

• << Previous: Study Types & Terminology
• Next: Assess for Quality and Bias of Studies >>
• Adelphi University Libraries
• Common Review Types
• Integrative Reviews
• Scoping Reviews
• Rapid Reviews
• Meta-Analysis/Meta-Synthesis
• Selecting a Review Type
• Types of Questions
• Key Features and Limitations
• Is a Systematic Review Right for Your Research?
• Guidelines for Student Researchers
• Training Resources
• Register Your Protocol
• Handbooks & Manuals
• Reporting Guidelines
• PRESS 2015 Guidelines
• Search Strategies
• Selected Databases
• Grey Literature
• Handsearching
• Citation Searching
• Screening Studies
• Study Types & Terminology
• Reducing Bias
• Quality Assessment/Risk of Bias Tools
• Tools for Specific Study Types
• Data Collection/Extraction
• Broad Functionality Programs & Tools
• Search Strategy Tools
• Deduplication Tools
• Screening Tools
• Data Extraction & Management Tools
• Meta Analysis Tools
• Books on Systematic Reviews
• Finding Systematic Review Articles in the Databases
• Systematic Review Journals
• More Resources
• Evidence-Based Practice Research in Nursing This link opens in a new window
• Citation Management Programs
• Last Updated: Aug 2, 2024 5:50 PM
• URL: https://libguides.adelphi.edu/Systematic_Reviews

Systematic review and narrative synthesis of the experiences of individuals with chronic pain participating in digital pain management interventions

• Strain, Justin Damian Russell
• Welch, Lindsay
• Sadler, Euan

Background The use of digital pain management interventions has grown since the Covid 19 pandemic. The aim of this study was to systematically review and synthesise evidence from qualitative studies regarding the experiences of individuals with chronic pain participating in digital pain management interventions in primary care and community settings. Methods Fourteen databases were searched, as well as citation tracking and hand-searching reference lists of included articles. The latest search was completed by 07/07/2023. Qualitative studies of patient and carer perspectives of digital pain management interventions for adults aged 18 and over with non-malignant chronic pain were included. All studies were appraised for quality using the Critical Appraisal Skills Programme Qualitative Checklist. A narrative synthesis approach was used to synthesise the findings. Normalisation Process Theory was used to understand how individuals with chronic pain make sense of digital pain management interventions and incorporate knowledge, skills and strategies learnt into their day-to-day lives. Results Eleven studies, encompassing both digital applications for use on smartphones/ mobile devices and user-directed online modular programmes, were included in the synthesis. Three main themes and related subthemes were identified from the included studies: 1) Making sense of the digital intervention (Subthemes: Tailoring to user's needs; Human contact and support; Accessibility of the digital intervention; Personal and environmental factors affecting engagement with digital interventions); 2) Initiating and Maintaining Behaviour Change (Subthemes: Planning activity; Being active); and 3) Personal Growth (Subthemes: Gaining understanding and skills; Gaining and acting on feedback; Negotiating a new relationship with pain). Conclusion Recommendations. The key recommendations from our findings are that digital pain management interventions should provide: Specific and tailored information for individual participants. Focus on changing attitudes and behaviours and reframing perceptions of pain. Structured goal setting with prompts to review goals. Potential healthcare professional support alongside the digital intervention. Limitations of the review. To reduce bias, it would have been preferable for more than one author to independently fully analyse each paper and to identify themes and sub-themes. Instead, the identified themes and sub-themes were discussed with two other authors in the team (ES, LW) to reach a consensus view on final themes and sub-themes. One author (JS) received a Research Internship and Research Initiation Award funded by NIHR Applied Research Collaboration (ARC) Wessex (https://www.arc-wx.nihr.ac.uk/) and NHS England (https://www.england.nhs.uk/). The protocol for this review was registered with the National Institute of Health Research (NIHR) PROSPERO international database for registering systematic reviews (PROSPERO Registration Number CRD42021257768).

This paper is in the following e-collection/theme issue:

Published on 30.7.2024 in Vol 26 (2024)

This is a member publication of University of Oxford (Jisc)

The Acceptability, Engagement, and Feasibility of Mental Health Apps for Marginalized and Underserved Young People: Systematic Review and Qualitative Study

Authors of this article:

• Holly Alice Bear 1 , DPhil   ; 
• Lara Ayala Nunes 1 , DPhil   ; 
• Giovanni Ramos 2 , PhD   ; 
• Tanya Manchanda 1 , MEd   ; 
• Blossom Fernandes 1 , PhD   ; 
• Sophia Chabursky 3 , MSc   ; 
• Sabine Walper 3 , DPhil   ; 
• Edward Watkins 4 , DPhil   ; 
• Mina Fazel 1 , DM  

1 Department of Psychiatry, University of Oxford, Oxford, United Kingdom

2 Department of Psychological Science, University of California, Irvine, CA, United States

3 German Youth Institute, Munich, Germany

4 School of Psychology, University of Exeter, Exeter, United Kingdom

Corresponding Author:

Holly Alice Bear, DPhil

Department of Psychiatry

University of Oxford

Warneford Hospital

Warneford Lane

Oxford, OX3 7JX

United Kingdom

Phone: 44 01865 6182

Email: [email protected]

Background: Smartphone apps may provide an opportunity to deliver mental health resources and interventions in a scalable and cost-effective manner. However, young people from marginalized and underserved groups face numerous and unique challenges to accessing, engaging with, and benefiting from these apps.

Objective: This study aims to better understand the acceptability (ie, perceived usefulness and satisfaction with an app) and feasibility (ie, the extent to which an app was successfully used) of mental health apps for underserved young people. A secondary aim was to establish whether adaptations can be made to increase the accessibility and inclusivity of apps for these groups.

Methods: We conducted 2 sequential studies, consisting of a systematic literature review of mental health apps for underserved populations followed by a qualitative study with underserved young male participants (n=20; age: mean 19). Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, an electronic search of 5 databases was conducted in 2021. The search yielded 18,687 results, of which 14 articles met the eligibility criteria.

Results: The included studies comprised a range of groups, including those affected by homelessness, having physical health conditions, living in low- and middle-income countries, and those with sexual and gender minority identities. Establishing and maintaining user engagement was a pervasive challenge across mental health apps and populations, and dropout was a reported problem among nearly all the included studies. Positive subjective reports of usability, satisfaction, and acceptability were insufficient to determine users’ objective engagement.

Conclusions: Despite the significant amount of funding directed to the development of mental health apps, juxtaposed with only limited empirical evidence to support their effectiveness, few apps have been deliberately developed or adapted to meet the heterogeneous needs of marginalized and underserved young people. Before mental health apps are scaled up, a greater understanding is needed of the types of services that more at-risk young people and those in limited-resource settings prefer (eg, standard vs digital) followed by more rigorous and consistent demonstrations of acceptability, effectiveness, and cost-effectiveness. Adopting an iterative participatory approach by involving young people in the development and evaluation process is an essential step in enhancing the adoption of any intervention, including apps, in “real-world” settings and will support future implementation and sustainability efforts to ensure that marginalized and underserved groups are reached.

Trial Registration: PROSPERO CRD42021254241; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=254241

Introduction

Addressing health inequities is a key challenge for the mental health field, especially when trying to ensure that interventions and services are accessible and acceptable for all populations. Nearly 50% of lifelong mental health disorders begin by the age of 14 years, and by the age of 24 years, 75% of mental health disorders have begun [ 1 ]. Given the frequent onset of mental health problems during youth, here defined as the period between 15 and 24 years, special attention must be paid to older adolescents, including those from underserved and marginalized minority groups and socioeconomically deprived backgrounds [ 2 ]. In these groups, common barriers to accessing mental health services can be exacerbated (eg, poor mental health literacy, lack of knowledge about where to seek help, negative attitudes toward professional help seeking, embarrassment, preference for self-reliance, fear of stigma, and confidentiality concerns), and additional barriers exist (eg, reliance on informal supports, shame, lack of housing or money, and therapist factors, such as different race and level of experience), creating increased risk of untreated mental health problems and thus poorer mental health outcomes [ 3 - 7 ]. In this study, marginalized and underserved populations are defined as those with higher prevalence of mental health problems and lower rates of help seeking, such as racially and ethnically minoritized individuals, rural and remote communities, financially deprived groups, individuals experiencing homelessness, refugee and migrant populations, and sexual and gender minority groups [ 8 - 10 ] and those with lower inclusion in mental health intervention research than one would expect from population estimates [ 11 ], respectively. These groups are exposed to risk factors for poor mental health and experience disparities in mental health care, including lower access to care, poorer treatment quality, and limited engagement in treatment [ 5 , 10 , 12 ].

Smartphone apps could offer an opportunity to deliver mental health and well-being resources and interventions in a scalable, cost-effective, and potentially personalized manner, particularly for those who experience the greatest barriers to accessing health care [ 13 , 14 ]. Given that smartphone ownership is nearly ubiquitous among young people in high-income nations and increasingly across lower-resource settings, apps have the potential to address some of the accessibility issues in service provision for young people’s mental health [ 15 ]. Young people are more digitally connected (ie, they are more likely to own smartphones and spend more time on the web) and more likely to seek health information on the web than older generations, meaning that app-based interventions may be particularly well-suited for this population [ 16 , 17 ].

Not surprisingly, the number of mental health apps being developed, both commercially and in academic research programs, has expanded rapidly, outpacing scientific evaluations of their effectiveness [ 18 , 19 ]. Emerging evidence suggests that some apps may produce significant symptom improvement across multiple outcomes, compared with waitlist or control conditions [ 20 - 22 ]. Despite promise, empirical research often fails to translate into meaningful and sustained implementation in “real-world” settings [ 23 , 24 ]. Research has focused primarily on efficacy under ideal “laboratory” conditions rather than effectiveness in real-world settings [ 25 ]. Therefore, at present, most apps, especially those available to the public, lack strong empirical support [ 19 ]. The acceptability (ie, perception that a given technology is useful, agreeable, palatable, or satisfactory); accessibility (ie, the technology being easy to obtain or use); engagement (ie, initial adoption and sustained interactions with the technology, including the level of app use, intervention adherence, and premature dropout); and feasibility (ie, the actual fit, utility, or suitability and the extent to which the technology can be successfully used or conducted within a given context) of apps for marginalized and underserved groups remain poorly understood [ 25 - 28 ]. Although mental health apps may provide a possible solution, marginalized and underserved groups of young people face unique challenges to engage with and benefit from these interventions (eg, intervention cost, content that is not culturally attuned and lack of reliable access to the internet) and are typically underrepresented in intervention research [ 25 , 27 ]. Although increased access is often seen as a major benefit of digital mental health interventions, issues related to the “digital divide” describe the phenomena that technology is not equally available to all social groups due to economic, social, or cultural inequalities and is a potential ethical concern [ 29 ]. Furthermore, underrepresentation in the intervention development process potentially reinforces structural inequalities by limiting the availability of products that are culturally accessible, inclusive, and effective or by skewing the product features to attract young people from more advantaged backgrounds [ 9 , 14 ]. Therefore, considering diversity, equity, and inclusion issues at the outset of health care research as well as within app evaluation is essential to prevent the perpetuation of existing inequities [ 27 ].

To date, little has been published on the attempts to create new or adapt existing app interventions to meet the heterogeneous needs of diverse groups of young people [ 9 , 14 ]. Moving forward, careful consideration is needed to ensure optimal leveraging of all mental health intervention research, including that of mental health apps, to increase health equity while also ensuring that innovations do not inadvertently widen the digital divide and exacerbate health inequalities [ 14 ]. Although the efficacy of many mental health apps remains unclear, future attempts to translate findings for underserved populations will need to ensure that all apps are developed with enough flexibility to fit a wider range of user needs and preferences. To achieve this goal, research is needed to assess the acceptability and feasibility of mental health apps for underserved young people to ensure that they are not further excluded from research and to advance toward mental health provision that meets their needs.

This Research

We conducted two sequential studies: (1) a systematic literature review and (2) a qualitative study with a targeted sample of young people who often are underrepresented in research, with limited access to health care and socioeconomic deprivation. The overarching aim of these combined studies was to better understand whether mental health apps are feasible and acceptable to underserved young people. A secondary aim was to determine which adaptations might enable accessibility of and effective engagement with mental health apps for these groups.

The research questions of interest were as follows: (1) On the basis of the existing literature, are mental health apps acceptable, feasible, and engaging for marginalized and underserved young people and how have these constructs been measured? (2) On the basis of the qualitative study, what are young people’s experiences of using a mental well-being app, including its acceptability, feasibility, and level of engagement? (3) On the basis of both studies, are apps an acceptable, feasible, and engaging intervention approach to meet the specific needs of underserved young people? What adaptations can be made to ensure that mental health apps are accessible and inclusive for these groups?

To fully address our research questions, we adopted a 2-pronged approach. First, we conducted a systematic review of the literature to better understand the acceptability and feasibility of mental health apps for underserved young people. To explore the findings of the systematic review in greater depth and to provide further insights from multiple perspectives, we next conducted a qualitative study with young men not in education, employment, or training (NEET) in the United Kingdom and Spain and asylum seekers and refugees in Germany. The interviews were conducted between August 2021 and February 2022.

Systematic Review

Literature search and search strategy.

An electronic literature search was performed in English on the following databases from January 2009 to May 2021: Cochrane Library, Embase, MEDLINE, and PsycINFO. We used key search terms relating to (1) underserved young people; (2) mental health mobile apps; and (3) acceptability, feasibility, and engagement. The search strategy was guided by similar reviews exploring digital mental health interventions for young people [ 25 , 30 ], and the terms for apps were derived from Cochrane reviews [ 31 , 32 ]. An updated search was conducted in September 2023. The full search strategies are available in Multimedia Appendix 1 .

Inclusion and Exclusion Criteria

Screened articles were included if (1) the study targeted marginalized and underserved young people with a mean age of 15 to 25 years, including individuals who were NEET, apprentices, teenage parents, members of minoritized racial or ethnic groups, members of sexual and gender minoritized groups, residents of low- and middle-income countries (LMICs), experiencing homelessness, socioeconomic deprivation, refugees or asylum seekers, and migrants; individuals with substance use disorders; those under state or statutory care; people with physical disabilities; and individuals involved in the criminal justice system or incarcerated; (2) the intervention was a “native” mobile app (ie, not on a web browser), whose primary aim was to promote well-being, prevent mental health problems, or treat existing mental health problems, delivered as a stand-alone intervention or as an adjunct to therapist-assisted interventions; (3) the primary outcome was a measure of mental health or well-being; and (4) the study reported a measure of user acceptability or feasibility.

Articles were excluded if (1) the mean age of participants was outside of the 15 to 25 years range; (2) the intervention was not a mobile app, that is, other digital interventions, including teletherapy (eg, therapy delivered by phone, SMS text messages, video platforms, or PCs); and (3) there was no measure of acceptability or feasibility. Gray literature was not included in the search.

Study Selection

In accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [ 33 ], the flowchart presented in Figure 1 provides step-by-step details of the study selection procedure. The PRISMA checklist is provided in Multimedia Appendix 1 . The search strategy identified 11,539 citations after deduplication. After an initial screening of the titles, which resulted in the exclusion of 10,061 (87.19%) irrelevant entries, the abstracts of 1478 (12.81%) studies were screened by 4 members of the review team (LAN, HAB, BF, and TM). The identified 176 (11.91%) full texts were then screened by LAN. In this final stage, 11 (6.2%) studies, corresponding to 9 interventions, were identified for inclusion in the review, with 8 (73%) found through the electronic search and 3 (27%) through manual searches of the reference lists of relevant articles. The updated search identified a further 7148 citations after deduplication, of which 3 met inclusion criteria and were included in the review.

Data Extraction

Data were extracted by 1 reviewer (LAN, HAB, or BF) and reviewed for accuracy and completeness by a second reviewer. After verifying all the extracted data, discrepancies were resolved by discussion or adjudication by another author (MF). Extracted data included information on study characteristics (ie, authors, publication year, country, study design, and study population); intervention characteristics (ie, characteristics of the technology, app name, therapeutic modality, and intervention outcomes); and feasibility and acceptability.

Quality Assessment

We used the Mixed Methods Appraisal Tool (MMAT; version 2018) to assess the methodological quality of the included studies [ 34 ]. MMAT was developed by combining the core relevant methodological criteria found in different well-known and widely used qualitative and quantitative critical appraisal tools. The MMAT consists of 2 screening questions applicable to all types of study design and a further 5 questions applicable to specific study designs. Responses were rated on a categorical scale as “no,” “unclear,” or “yes” to any of the methodological quality criteria. Quality assessments were made by 1 reviewer (TM). We did not exclude any studies based on quality assessment scores.

Data Synthesis and Analysis

The extracted data were collated and summarized to produce a narrative summary of the study; sample characteristics; and acceptability, feasibility, and engagement outcomes. A codebook approach was used to code and synthesize implementation data from all available sources according to the outcome categories [ 35 ].

Qualitative Study

Study context.

The Emotional Competence for Well-Being (ECoWeB) cohort multiple randomized controlled trial involved a longitudinal prospective cohort to examine the well-being, mental health, and emotional competence in individuals aged 16 to 22 years across 12 months. The experimental arm was an emotional competence self-help app (ie, MyMoodCoach).

Intervention

MyMoodCoach was designed to test if an app could improve different processes affecting mental well-being, including, but not limited to, improving emotion regulation by reducing maladaptive strategies such as worry and rumination and replacing them with constructive alternatives and problem-solving and enhancing emotional knowledge and perception through psychoeducation and learning tasks. The app was designed for young people broadly and was not targeted at a specific population. Full details of the ECoWeB trial are reported in the study protocol [ 36 ].

Participant Recruitment

In parallel to the trial, we additionally recruited young people to understand the views and experiences of those underrepresented in the study sample (and most other app-based studies). As there was an overrecruitment of White, university-educated female participants in the study sample, we decided to only recruit male participants and to focus on 2 specific groups: NEET men and migrant populations (including both voluntary and forced migrants). In the United Kingdom and Spain, recruitment was conducted through a variety of channels, including Twitter, targeted adverts on social media (eg, Instagram and Facebook), newsletters sent to youth and practitioner networks, outreach to third sector organizations and mental health support groups, and advertisements placed on university and charity websites. In Germany, participants were recruited directly from refugee homes, integration courses, migration services, and youth centers.

Inclusion criteria required the participants to be (1) aged between 16 and 22 years; (2) able to speak and read English, Spanish, or German; (3) male; (4) NEET (United Kingdom), NEET or migrant (Spain), or asylum seeker or refugee (Germany); (5) having access to a smartphone with the minimal technological specifications necessary for the app (ie, iOS 9 or later or Android 8.0 or later); and (6) having access to the internet via mobile data or Wi-Fi. The exclusion criteria were having current suicidal ideation, psychosis, or bipolar disorder.

The research team was led by the principal investigator, MF, professor of Adolescent Psychiatry and Consultant in Children’s Psychological Medicine, with particular expertise in the mental health needs of refugee populations. EW, professor of Experimental and Applied Clinical Psychology, Chartered Clinical Psychologist, provided additional supervisory input and guidance as a leading expert in the field of child and adolescent mental health research. SW, professor of Education with a focus on youth research and director at German Youth Institute, provided supervisory oversight of the interviews in Germany. HAB and LAN were postdoctoral researchers at the time the research was conducted, both of whom have several years of experience conducting qualitative research with young people and expertise in analyzing qualitative data. SC, researcher at the German Youth Institute, conducted and analyzed the interviews in Germany and has experience conducting qualitative research with refugee populations. The authors had no relationship with any of the participants.

In the United Kingdom and Spain, participation involved (1) short questionnaires about mood and feelings and demographic questions such as age, gender, race, ethnicity, educational attainment, and country of origin; (2) downloading MyMoodCoach and using it for 4 weeks; and (3) completing a follow-up interview. Interviews were conducted by HAB and LAN via MS Teams (Microsoft, Corp) and lasted approximately 45 minutes. A similar procedure was followed in conducting the interviews with refugees in Germany, but given the likely language barriers in navigating the app, an additional, earlier interview was included 2 to 3 weeks after the initial instructions had been sent to explore if features of the app were understood and to clarify questions. After 2 further weeks, a second interview was conducted. One interview was conducted in person, while all the others were conducted over the telephone.

Interviews followed a semistructured schedule based on a taxonomy of implementation constructs [ 26 ]. Topics included self-reported app use, satisfaction and feedback about content, usability, and acceptability ( Multimedia Appendix 2 ). In Germany, the interview topic guide was translated and adapted to the target group. As not to stigmatize the young men, the wording “refugee” was avoided, and instead, when referring to the target group, the wording “young man such as yourself” was used.

Ethical Considerations

In the United Kingdom, ethical approval was granted by the University of Exeter Research Ethics Committee (eCLESPsy000048 v10.0); in Spain, by the Jaime I University Research Ethics Committee (CD/93/2021); and in Germany, by the Ethics Committee of the Faculty of Medicine at Ludwig-Maximilians University Munich (PNr 19-0468/19-0315). Cognizant of the ethical and practical implications of conducting research with underserved populations, we conducted the interviews in private, quiet spaces and in a friendly and reassuring manner to ensure that participants felt comfortable and safe.

With prior consent, all interviews were audio recorded. Participants were reimbursed with a shopping voucher of up to €50 or £50 (US $64) for their time and app use.

Analytic Strategy

Interviews were transcribed verbatim, and the transcripts were assigned a unique pseudonym to anonymize participants. The interviews were analyzed using a combination of theory- and data-driven analysis techniques, consisting primarily of deductive, theory-driven thematic analysis [ 37 ]. Analysis of the transcripts in the United Kingdom and Spain was conducted by HAB, using NVivo (version 11; Lumivero). Following a similar procedure, the interviews in Germany were transcribed and analyzed using the coding software MAXQDA (VERBI GmbH) by SC.

Initial familiarization with the data was achieved through the transcription process and iterative rereading of the interviews. Analysis was carried out through a recursive process of open coding, when concepts were named and their properties and dimensions identified, followed by axial coding, when links and associations were drawn between codes. Codes were based on language used by the young people and were applied to each new unit of meaning. Data extracts were multiply coded when appropriate, as were contradictory and minority features of the data. The data set was iteratively reviewed, and codes were systematically applied to the whole data set until a finalized coding manual was established. Codes were organized into potential themes using thematic maps and tables. The development of the coding manual was iteratively reviewed and refined through discussion with all authors throughout the analysis process to ensure the reliability and rigor of the process and results.

Approach to Inquiry

Analysis was conducted from a critical realist perspective to provide a more nuanced understanding and explanation of participants’ experiences [ 38 ]. This position assumes that although participants’ accounts provide important insights about the real world, these accounts are not objective and represent an interpretation of reality [ 39 ]. These data require interpretation and explanation by the researcher, who also has their own perspectives on the world, to better understand the underlying mechanisms and processes and, in turn, make recommendations for practice [ 40 ].

Study Characteristics

Characteristics of the 14 included studies, examining 12 interventions, are presented in Table 1 . The included studies were published in the United States (n=3), Australia (n=3), Switzerland (n=2), India (n=2), South Korea (n=1) and Germany (n=1). As for the type of intervention, 7 apps were stand-alone, and the rest (n=5) were delivered in combination with other forms of professional support. The interventions (n=12) were targeted at apprentices and the unemployed (n=3), homeless populations (n=2), those with physical health conditions (n=2), sexual and gender minoritized individuals (n=3), those residing in LMICs (n=1) and those with co-occurring autism spectrum disorders (n=1). Most of the interventions had been co-designed with young people (11/12, 92%). The study sample size ranged from 9 to 877, with 4 of the 14 studies having a sample size of >200 participants.

a CBT: cognitive behavioral therapy.

b CF: cystic fibrosis.

c JIA: juvenile idiopathic arthritis.

d T1D: type 1 diabetes.

e RCT: randomized controlled trial.

f iCBT: internet-based cognitive behavioral therapy.

g Mobile phones were preloaded with several apps designed to promote mental health wellness and provide real-time resources. Pocket Helper was 1 app specifically designed for this study.

h Studies identified in the updated search.

Study Quality

The included studies varied in their methodological quality ( Multimedia Appendix 3 ) [ 41 - 54 ]. Most (13/14, 93%) were judged to contain possible limitations in at least 1 criterion. All studies but 1 (13/14, 93%) were clear in their description of study participants or the process of recruiting a sample representative of the population of interest. All studies addressed the research question using collected data and reported in some way on feasibility and acceptability. Most studies (12/14, 86%) effectively used appropriate qualitative, quantitative, or mixed methods to answer their research question. However, many studies (6/14, 43%) did not have a sufficiently large sample to warrant definitive conclusions about the feasibility and acceptability of the intervention studied.

Acceptability and Feasibility

User acceptability was found to be high across all included studies, with participants rating the apps positively and reporting high satisfaction with the content of the interventions ( Table 2 ). In studies where participants were asked to indicate if they would recommend the study to someone else, the vast majority reported they would [ 42 , 44 - 46 , 51 ]. It is notable that despite many users across the studies reporting high satisfaction levels and being willing to recommend the apps, they themselves did not intend to continue using the apps (ie, low predicted engagement), as they did not find them useful or relevant for their own circumstances [ 41 , 42 , 51 ]. It is also important to note that many studies incentivized participation with payments, prize draws, and vouchers, and this may have influenced acceptability ratings and engagement [ 41 - 43 , 46 , 49 - 51 , 53 , 54 ]. Furthermore, in the only study that asked if participants would pay to use apps, most were unwilling to do so [ 42 ].

a Studies identified in the updated search.

Regarding co-design strategies used in these studies, early stakeholder consultation and service provider focus groups were conducted in the early development phase of POD Adventures, a gamified intervention for people with or at risk of anxiety, depression, and conduct difficulties in India; the results highlighted that “self-help” was not a culturally congruent concept for most Indian adolescents [ 47 ]. This early feedback was important as it revealed the norms around seeking or receiving direct instruction from parents, teachers, and other elders and that support from a counselor might be necessary to ensure acceptability, feasibility, and engagement [ 47 ]. The app was therefore designed to incorporate a combination of teaching methods, including direct instruction, modeling, and practice to accommodate different learning styles and to emphasize self-efficacy [ 47 ]. Furthermore, user testing also highlighted the need for more direct language, particularly around problem-solving concepts [ 47 ]. The iterative study methodology used in this study enabled the participants to guide the development and provide their inputs at each stage to increase acceptability and feasibility.

Although, overall, the apps were well received by young people, poor engagement (eg, not engaging at the recommended frequency or complete the full course of the intervention), measured through both self-report, intervention adherence, and data capture was a commonly reported issue. Many studies failed to achieve continued participation, with high rates of attrition [ 41 , 42 , 44 ]. In addition, app use often decreased markedly after the initial few weeks [ 41 , 42 , 44 ]. The results of some studies suggested that engagement (in the form of intervention completion) was related to the effectiveness the intervention [ 41 , 42 ]. Although engagement was problematic in many of the stand-alone interventions, engagement and study participation in a school setting seemed more promising [ 49 ]. For example, a proactive invitation for study participation in a school enabled 4 out of 5 eligible adolescents to participate in the “ready4life” life skills program [ 49 ]. This strategy consisted of individuals who were trained in the program to be delivered, giving arranged sessions lasting 30 minutes in participating vocational schools during regular school lessons reserved for health education. Within this session, the students were informed about and invited to participate in the study, including being informed about the study’s aims and assessments, reimbursement, and data protection.

Barriers to Engagement

Qualitative interviews and user feedback provided important insights about relevant barriers to engagement. The most frequently mentioned reasons for not using the app were that participants could not see the obvious benefits of using the app [ 41 ], lack of time or forgetting [ 42 ], and technical difficulties [ 41 , 50 , 51 ]. In a life skills training app for vocational students, participation was lower in male adolescents and among those reporting an immigrant background [ 49 ], although the reasons behind this poor engagement remained unclear.

What Do Young People Want From Apps?

There was some heterogeneity between studies in terms of the features and content that participants found acceptable and appropriate. For example, findings suggested that young people who experience homelessness tended to prefer both automated and self-help features compared with ones involving more direct human interaction [ 46 ]. However, participants in other studies valued both human interaction with professionals either via the app interface or through face-to-face contact and self-help features [ 47 ]. Human support was suggested as being helpful in offering both instruction and guidance as well as personalized support when needed. Numerous participants wanted opportunities to interact with peers [ 43 , 44 , 52 ] and even suggested connecting apps to social media [ 41 ]. Others also wanted the design of the apps to be more attractive (eg, improve the layout and create a more intuitive structure) and made suggestions about how gamifying apps could make them more interesting [ 41 , 43 , 47 ].

We interviewed 13 young men in the United Kingdom (age: mean 18.7, SD 2.5 y), 2 in Spain (age: mean 17, SD 0 y), and 5 in Germany (age: mean 20.2, SD 1.6 y). In the United Kingdom, 62% (8/13) of the participants self-reported as ethnically White, compared with 50% (1/2) in Spain and 20% (1/5) in Germany ( Table 3 ).

In terms of participants’ mental health and well-being ( Table 4 ), the mean Patient Health Questionnaire-9 score in the United Kingdom was 9.7 (SD 7.3) compared with 5 (SD 1.4) in Spain.

a WEBWBS: Warwick-Edinburgh Mental Well-Being Scale.

b Not available.

c PHQ-9: Patient Health Questionnaire-9.

d GAD-7: Generalized Anxiety Disorder Assessment.

A key finding was that despite best efforts and financial incentives, recruiting underserved young male participants, especially in Spain and Germany, was challenging. This might suggest that these young people may not deem such an emotional competence app as relevant or useful to them, making recruitment and engagement problematic. We also assessed if the app was deemed acceptable (ie, useful, agreeable, palatable, or satisfactory) and appropriate (ie, relevant, suitable, or compatible). Overall, the app was viewed by participants in the United Kingdom, Spain, and Germany as being appropriate and relevant for young people of different ages and walks of life, as they thought that all young people had a smartphone and were adept at using technology:

So, I was able to learn about my feelings, I was able to evaluate how I actually felt today, concerning my feelings, if I was angry or I was sad. I was actually able to write them down in detail. [Participant in Germany]

Several participants commented that the content of the app was best suited to university and school students. Another common view was that the app was better suited to those struggling with their mental health and that it was less relevant for those for whom things were going well. Many participants perceived the app to be aimed at improving mental health problems, as opposed to being a universal intervention intended to improve well-being, which represented a barrier to engagement. Of those who reported that the app was not relevant to them, they did see it as being of potential use to friends and family members who were stressed, anxious, or going through a difficult time:

There will be folks who maybe aren’t going through a good time in their lives, and they will need the app to feel... to understand themselves, mostly. And I think it’s relevant at any age, because I am lucky that I don’t think I need it as much as someone else who feels like that. [Participant in Spain]

Partly it was important, partly it was not. I’ll give an example again, for example if a refugee came to Germany from a war zone, it’s going to be difficult, very difficult to find a topic that would fit him, for the future I mean, so the version now is already okay if you want all persons to use this app. Partly it’s already relevant and partly it’s not. If someone has mental problems or bad experiences, you cannot find such a topic in the app. [Participant in Germany]

Although some participants reported using the app regularly during the 4-week study period, a consistent finding was that participants tended to use the app most when they first downloaded it, with a marked reduction in use over time:

Uh, I probably used it about three times in the first week. And then not really that much at all I’m afraid. [Participant in the United Kingdom]

I don’t know, I just dropped off using a little bit after a couple of weeks, but I’ve been trying to keep on top of doing that like the daily rating things and everything.... I kind of lost my motivation to use it. [Participant in the United Kingdom]

We identified several barriers that hindered participants’ engagement and use of the app. These included the following: (1) repetitive and time-consuming app contents, (2) a paucity of new content and personalized or interactive tools (eg, matching mood to tools), (3) unclear instructions, (4) a lack of rationale for the app, (5) perceiving the app as not being relevant, (6) a lack of motivation, and (7) privacy concerns:

Yes, for example, I would not like to write in this diary, because I do not know if it would be one hundred percent anonymous and if others might read it. And maybe I have more privacy if I do not write it. [Participant in Germany]

I think by now I would slowly stop using the app. It was nice up to this point, but I think for me I might need a step further now. To really deal with my personal problems and I don’t know how much an app like this can help and that rather an expert and therapy is needed. [Participant in Germany]

For the asylum seekers and refugees in Germany, the language and content of the app was not suited to their needs. The participants would have preferred the app in their native language as some had to use translation programs to help access the content. Furthermore, specific topics of relevance to refugees were missing, such as dealing with asylum uncertainty, whereabouts of family members, and their living situation.

Finally, underserved young people, including asylum seekers and refugees, migrants, and those NEET, are more likely to experience financial deprivations and therefore less likely to pay directly for apps, especially for those that do not address their primary difficulties:

If it came to the point that I had to pay for it, I would look for free options. [Participant in Germany]

Summary of Findings

The use of mobile apps in mental health care continues to attract interest and investment; however, research geared toward understanding the needs of marginalized and underserved populations is still nascent. This study, focusing on the implementation of mental health apps in underserved young people, highlighted that little research exists to support the widespread adoption of these apps as a mental health intervention for marginalized and underserved groups. Findings from both our systematic review and qualitative study were largely consistent: markers of acceptability and usability were positive; however, engagement for underserved young people was low, which is notable given the widespread ownership of smartphones [ 55 , 56 ]. To date, research has focused primarily on efficacy studies rather than effectiveness and implementation in “real-world” settings and may have overestimated users’ “natural tendency” to adopt smartphone apps for their mental health and well-being [ 57 ]. Our findings suggest that despite the rapid proliferation of mobile mental health technology, the uptake and engagement of mental health apps among marginalized young people are low and remain a key implementation challenge.

Our data suggest that establishing and maintaining user commitment and engagement in the content of the intervention as intended is a pervasive challenge across mental health apps and marginalized populations, and premature dropout was prominent in nearly all the included studies. This is consistent with the literature that suggests that the majority of those offered these app-based interventions do not engage at the recommended frequency or complete the full course of treatment [ 58 , 59 ]. In this study, various app components were associated with engagement level, with the most engaging interventions providing young people with some form of associated real-human interaction and those having a more interactive interface. This aligned with other findings that the feedback of personalized information to participants is an especially important aspect of creating engaging and impactful digital tools [ 60 ]. Young people tend to quickly disengage if there are technical difficulties or if the app does not specifically target their perceived needs [ 41 , 50 , 51 ]. Furthermore, recruitment of marginalized groups to app-based studies is difficult. For instance, in this study, the use of advertisements, financial incentives, vouchers, and prize draw incentives seemed to be insufficient to recruit a significant number of participants in Spain and Germany.

Measuring engagement is a challenge that has likely contributed to our lack of knowledge on app components that effectively increase user engagement. Reporting engagement with mental health apps in intervention trials is highly variable, and a number of basic metrics of intervention engagement, such as rate of intervention uptake, weekly use patterns, and number of intervention completers, are available, yet not routinely reported [ 58 , 59 ]. The results of this study highlight the importance of objective engagement measures and that relying on positive subjective self-reports of usability, satisfaction, acceptability, and feasibility is insufficient to determine actual engagement. Furthermore, the findings suggest that apps involving human interactions with a professional (eg, therapist or counselor) or that are completed in a supervised setting tend to be more acceptable and effective and have higher engagement rates [ 47 , 48 ]. Our research suggests that similar to traditional face-to-face mental health services, app-based programs still face numerous barriers to reach marginalized youth, especially since the mental health apps available to the public do not seem to consider the unique developmental needs of these groups, participants do not seem to perceive an obvious benefit from using them, and some potential users prefer to interact with a professional face to face. Thus, it is also possible that the digital mental health field might be inadvertently contributing to mental health inequities among this population by not engaging marginalized groups sufficiently at the outset of research to ensure that the designed app meets their needs. However, for the studies included in this study that did engage these groups in the co-design of the apps, there was no notable improvement in engagement. Thus, we hope these findings encourage researchers and clinicians to think more critically of the role that mental health apps can truly have in addressing mental health equities among underserved groups.

As in other areas of mental health research, young people from LMICs were underrepresented in these studies, which typically originated from high-income settings, including the United States, Australia, and Canada. There are relatively few app-based interventions that were designed or adapted for young people in LMICs that have been rigorously evaluated or are even available in local languages [ 47 , 48 ]. Many living in LMIC regions, for example, adults in Asian countries, are often faced with apps that are not culturally relevant or in the right language [ 61 ]. These inequities are surprising given the high rates of smartphone use in Asia, even in rural regions [ 62 ]. Yet, it is still likely that youth in this region faced barriers related to data availability and more limited phone access, which will likely inhibit the broad implementation of apps beyond research studies [ 16 ]. Considerable work is required to ensure the availability of mental health apps that fit a wide range of user needs and preferences. It is important to ensure that the acceptability and feasibility of mental health apps for young people residing in LMICs are prioritized so that they are not further excluded from relevant mental health research.

Finally, a significant challenge is the lack of diversity in mental health app research participation, which limits our understanding of “real-world” efficacy and implementation for underserved and marginalized groups. While undoubtedly invaluable, and indeed deemed gold standard when evaluating efficacy of interventions, randomized controlled trial of mental health apps are not without flaws [ 63 , 64 ]. Trial recruitment is often highly selective due to stringent inclusion and exclusion criteria resulting in lower inclusion in research than one would expect from population estimates [ 65 ]. In the United Kingdom, the National Institute for Health and Care Research data have revealed that geographies with the highest burden of disease also have the lowest number of patients taking part in research [ 66 ]. The postcodes in which research recruitment is low also aligns closely to areas where earnings are the lowest and indexes of deprivation are the highest [ 66 ]. There are many reasons why some groups are underrepresented in research: language barriers, culturally inappropriate explanations, poor health literacy and the use of jargon, communication not being suitable for people with special learning needs, requirement to complete many administrative forms, negative financial impact in participating, lack of effective incentives for participation, or lack of clarity around incentives, and specific cultural and religious beliefs [ 66 ]. Failing to include a broad range of participants is problematic in that results may not be generalizable to a broad population.

Limitations

Although this research was carefully executed and used a robust methodological approach with an exhaustive search strategy, it is not without limitations. Foremost, although the systematic review attempted to identify and include as many articles as possible, some papers may have been missed because of the inconsistencies in how feasibility and acceptability outcomes are recorded and reported. It was also difficult to ensure that all apps for this age group were identified because those aged between 15 and 25 years are harder to differentiate in adolescent and adult studies, meaning we might have missed some relevant studies where data could not be disaggregated by age. The exclusion of gray literature (eg, institutional reports and websites) may have also made us overlook potentially relevant apps, albeit lacking the quality assurance of peer-reviewed research. It is also likely that commercial organizations, including app companies, collect rich user demographic and engagement data but do not share it publicly, thus limiting our ability to conduct empirical analyses about the “real-world” acceptability, engagement, and implementation for specific populations. We did not analyze the extent to which publication bias may have influenced the results of our search, and, therefore, there may be a much higher number of mental health apps that have been developed with an underserved sample of young people, but due to their lack of efficacy or acceptability, these studies have not been submitted or accepted for publication. The sample sizes of many of the included studies were relatively low, which potentially limits their generalizability. However, we included all study designs so as to ensure that our learning from existing research was maximized. Furthermore, many of the studies included in the systematic review, as well as our qualitative study, had some form of language competency as an inclusion criterion (eg, English speaking), which likely excludes important perspectives from the results. For the qualitative study, we were only able to gather data from those who had used the app at least once and who were therefore somewhat engaged in the app. Despite our best efforts, we were unable to recruit participants who, following consent, had never then downloaded or used the app and so could not explore barriers to engagement for the least engaged young people or understand why the app was not appealing to those who chose not to proceed or take part. Those who did participate in this research were financially incentivized to do so and often highlighted the importance of this incentive in keeping them engaged. Therefore, we were unable to draw conclusions about the naturalistic engagement, feasibility, and acceptability of the app, if it were to be made available without payment in schools, universities, and health services or to be made commercially available on the app marketplace. It is also possible that social desirability bias (ie, a tendency to present reality to align with what is perceived to be socially acceptable) occurred during the interviews, whereby participants responded to the interview questions in a manner that they believed would be more acceptable to the study team, concealing their true opinions or experiences [ 67 , 68 ]. As previously noted by others, results may be subject to further bias in that findings could be led by more articulate young people, while it is more difficult to hear the voices of those who are less articulate or digitally literate [ 69 ]. Finally, it is also possible that the positionality of the research team, including our own experiences, backgrounds, and biases, impacted what information participants disclosed to the research team as well as the interpretation of the qualitative data in this study.

Recommendations of Adaptations to Increase Acceptability, Feasibility, and Engagement

To overcome this complex engagement and implementation challenge, we have taken together our findings with relevant previous literature to generate 3 key suggestions about how to improve the feasibility and potential utility of apps for young people from marginalized and underserved populations.

Increasing Participant Diversity in Mental Health Intervention Research

Studies should aim to prioritize the inclusion of marginalized groups in trials testing the effectiveness of digital interventions by intentionally planning recruitment efforts aimed to reach these communities [ 70 ]. First, steps can be taken to build trust, connections, and credibility between the research team and these communities. NHS England [ 66 ] suggests involving representatives from those groups during the inception and implementation of recruitment efforts. This approach ensures that the intervention is relevant to the target group by meeting their preferences and needs, incorporating culturally salient factors relevant for recruitment efforts, addressing concerns about community mistrust and participant resource constraints, and establishing partnerships with key community stakeholders that can be gatekeepers in the community [ 14 , 71 ]. These strategies are likely to improve research accessibility, recruitment, and retention. Research teams need to ensure that the findings and any actionable takeaways from the research conducted with the participants are shared with them by asking participants how they would like to receive this information (eg, verbal, written, or via a trusted advocate). Equally important is to explain that the research process can be slow. These steps help create a positive legacy for the research project and build trust between individuals and public institutions, helping future health researchers to further address underrepresentation of marginalized groups in digital research.

Identifying and Addressing Needs and Preferences of Underserved and Marginalized Groups by Using Human-Centered Design Principles

A comprehensive understanding of the needs, challenges, and life circumstances of the target population is a key implementation driver for designing relevant, engaging, and effective mental health apps. This knowledge is particularly important when the app is a stand-alone intervention received during daily life outside of traditional psychotherapy or human support [ 50 ]. This goal can be best achieved through a participatory approach, which reflects a growing recognition among intervention researchers and developers that end users need to be involved in the creation of interventions and their future iterations [ 47 , 72 ]. This process may involve a series of stages, including (1) person-centered co-design to ensure that tools are developed to be acceptable to the underserved or marginalized populations as well as meet their specific needs, life circumstances, and cultural norms [ 47 ]; (2) iterative testing that incorporates users’ feedback on a rolling basis to ensure the relevance of the intervention [ 43 , 47 , 72 ]; and (3) changes and adaptations needed to meet users’ needs in “real-world” settings including consideration of economic viability and implementation [ 27 ].

Especially relevant for the underserved and marginalized groups is the need (or lack thereof) to culturally adapt app interventions for specific racial, ethnic, or cultural groups through this person-centered design. In traditional face-to-face interventions, some have argued that all treatments need to be culturally adapted to ensure their validity, relevance, and effectiveness since these interventions are often developed with individuals who can be substantially different from some marginalized populations [ 73 ]. Similar to culturally adapted face-to-face interventions [ 74 - 76 ], culturally adapted digital mental health interventions seem to be effective [ 77 , 78 ]. However, there is no evidence that these culturally adapted interventions outperform the original programs [ 79 , 80 ]. Given that culturally adapting digital interventions is a time-consuming and resource-intensive process, this approach may not be sustainable and limit the dissemination and implementation impact of app programs [ 28 ]. In lieu of culturally adapting digital interventions without careful consideration, Ramos and Chavira [ 28 ] recommend using information gathered through person-centered approaches to integrate culture into the use of already available digital interventions (including apps), using an idiographic, flexible, and personalized approach. This strategy may have a broader implementation and dissemination potential, given that few researchers and clinicians are in a position to develop new apps.

Embedding Apps Within Existing Care Structures

Several systematic reviews and meta-analyses have demonstrated that app-based mental health interventions with a human-support component are more effective and more acceptable than stand-alone, fully automatized, or self-administered apps [ 13 , 25 , 81 ]. Young people seem to want practical skills and usable tools to apply to their current daily life stressors to improve their well-being and functioning. Intervention engagement is enhanced if the intervention serves an obvious purpose, is relevant, and has a clear rationale and instructions, and embedding these interventions within the systems and structures that are already working with users (eg, clinical services, schools, universities, and community agencies) will likely improve implementation. Considering the broad and highly varied nature of intervention formats and modalities, it may be useful for future research to focus on identifying core components of app-based interventions (ie, active ingredients of interventions associated with uptake, adherence, and clinical outcomes) that will allow such integration of app interventions into the varied context of care for marginalized youth.

Conclusions

Despite the enthusiasm that has surrounded the potential of digital technologies to revolutionize mental health and health care service delivery, little evidence yet supports the use of mental health apps for marginalized and underserved young people. Despite the substantial financial and human investment directed to the development of mental health apps over several years, only a small proportion have empirical evidence to support their effectiveness, and there have been few attempts to develop or adapt interventions to meet some of the more unique and heterogeneous needs of diverse groups of young people. Although acceptability seems to be good, engagement is poor and attrition is high, particularly if not supported by in-person elements. Given that most interventions are implemented in high-income countries, very little is known about the generalizability of the findings to LMICs and to a range of adolescents and young people with different socioeconomic, cultural, and racial backgrounds. In this paper, we have drawn several insights about the feasibility and acceptability of mental health apps for underserved young people that may be useful to future app-based mental health promotion and treatment projects. However, before the widespread adoption and scaling-up of digital mental health interventions progresses further, especially for more vulnerable and underserved populations and in settings with limited resources, a greater understanding is needed on the unique barriers faced by these groups in accessing treatment and the types of services young people themselves prefer (eg, standard vs digital) followed by more rigorous and consistent demonstrations of feasibility, effectiveness, and cost-effectiveness.

Acknowledgments

This project received funding from the European Union’s Horizon 2020 research and innovation program (grant agreement number 754657).

The authors are grateful to the young people who took the time to participate in this research and who shared their insights with us. The authors would also like to thank those who supported this research including professional youth advisor Emily Bampton, research assistant Catherine Reeve, and researchers Dr Alexandra Langmeyer and Simon Weiser. Finally, the authors would like to thank the ECoWeB (Emotional Competence for Well-Being) Consortium for their support and feedback throughout the duration of this research, including, but not limited to, Dr Lexy Newbold, Dr Azucena Garcia Palacios, and Dr Guadalupe Molinari.

Data Availability

The data extracted to support the findings of the systematic review are available from the corresponding author upon reasonable request. Due to the confidential and sensitive nature of the interview transcripts, qualitative data will not be made available.

Authors' Contributions

HAB, LAN, and MF designed the systematic review including the research questions and methods. LAN carried out the database search. HAB, LAN, TM, and BF conducted the study screening and data extraction. TM did the study quality assessments, and HAB did the data synthesis and analysis. MF, SW, EW, and HAB were involved in the conception of the qualitative study. HAB, LAN, and SC conducted the quality study including conducting the qualitative interviews and analysis. HAB wrote the first draft and HAB, LAN, MF, and GR contributed substantially to manuscript drafting. All authors contributed to the manuscript and approved the submitted version.

Conflicts of Interest

None declared.

Search strategy.

Topic guide.

Study quality assessment.

• Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. Jun 2005;62(6):593-602. [ CrossRef ] [ Medline ]
• Solmi M, Radua J, Olivola M, Croce E, Soardo L, Salazar de Pablo G, et al. Age at onset of mental disorders worldwide: large-scale meta-analysis of 192 epidemiological studies. Mol Psychiatry. Jan 2022;27(1):281-295. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brown A, Rice SM, Rickwood DJ, Parker AG. Systematic review of barriers and facilitators to accessing and engaging with mental health care among at-risk young people. Asia Pac Psychiatry. Mar 03, 2016;8(1):3-22. [ CrossRef ] [ Medline ]
• Marrast L, Himmelstein DU, Woolhandler S. Racial and ethnic disparities in mental health care for children and young adults: a national study. Int J Health Serv. Oct 20, 2016;46(4):810-824. [ CrossRef ] [ Medline ]
• Cook BL, Trinh NH, Li Z, Hou SS, Progovac AM. Trends in racial-ethnic disparities in access to mental health care, 2004-2012. Psychiatr Serv. Jan 01, 2017;68(1):9-16. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rimes KA, Shivakumar S, Ussher G, Baker D, Rahman Q, West E. Psychosocial factors associated with suicide attempts, ideation, and future risk in lesbian, gay, and bisexual youth. Crisis. Mar 2019;40(2):83-92. [ CrossRef ] [ Medline ]
• Reiss F. Socioeconomic inequalities and mental health problems in children and adolescents: a systematic review. Soc Sci Med. Aug 2013;90:24-31. [ CrossRef ] [ Medline ]
• Warnecke RB, Oh A, Breen N, Gehlert S, Paskett E, Tucker KL, et al. Approaching health disparities from a population perspective: the National Institutes of Health Centers for Population Health and Health Disparities. Am J Public Health. Sep 2008;98(9):1608-1615. [ CrossRef ]
• Schueller SM, Hunter JF, Figueroa C, Aguilera A. Use of digital mental health for marginalized and underserved populations. Curr Treat Options Psych. Jul 5, 2019;6(3):243-255. [ CrossRef ]
• Introduction: NIH minority health and health disparities strategic plan 2021-2025. National Institutes of Health National Institute on Minority Health and Health Disparities. URL: https://www.nimhd.nih.gov/about/strategic-plan/nih-strategic-plan-directors-foreword.html [accessed 2024-06-29]
• Improving inclusion of under-served groups in clinical research: Guidance from INCLUDE project. National Institute for Health and Care Excellence. Aug 7, 2020. URL: https:/​/www.​nihr.ac.uk/​documents/​improving-inclusion-of-under-served-groups-in-clinical-research-guidance-from-include-project/​25435 [accessed 2024-06-29]
• Evans-Lacko S, Aguilar-Gaxiola S, Al-Hamzawi A, Alonso J, Benjet C, Bruffaerts R, et al. Socio-economic variations in the mental health treatment gap for people with anxiety, mood, and substance use disorders: results from the WHO World Mental Health (WMH) surveys. Psychol Med. Nov 27, 2017;48(9):1560-1571. [ CrossRef ]
• Hollis C, Falconer CJ, Martin JL, Whittington C, Stockton S, Glazebrook C, et al. Annual research review: digital health interventions for children and young people with mental health problems - a systematic and meta-review. J Child Psychol Psychiatry. Apr 10, 2017;58(4):474-503. [ CrossRef ] [ Medline ]
• Friis-Healy EA, Nagy GA, Kollins SH. It is time to REACT: opportunities for digital mental health apps to reduce mental health disparities in racially and ethnically minoritized groups. JMIR Ment Health. Jan 26, 2021;8(1):e25456. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mobile fact sheet. Pew Research Center. Jan 31, 2024. URL: https://www.pewresearch.org/internet/fact-sheet/mobile/ [accessed 2024-07-12]
• Silver L. 2. In emerging economies, smartphone adoption has grown more quickly among younger generations. Pew Research Center. Feb 5, 2019. URL: https:/​/www.​pewresearch.org/​global/​2019/​02/​05/​in-emerging-economies-smartphone-adoption-has-grown-more-quickly-among-younger-generations/​ [accessed 2024-07-12]
• Pretorius C, Chambers D, Coyle D. Young people's online help-seeking and mental health difficulties: systematic narrative review. J Med Internet Res. Nov 19, 2019;21(11):e13873. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Grist R, Porter J, Stallard P. Mental health mobile apps for preadolescents and adolescents: a systematic review. J Med Internet Res. May 25, 2017;19(5):e176. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lehtimaki S, Martic J, Wahl B, Foster KT, Schwalbe N. Evidence on digital mental health interventions for adolescents and young people: systematic overview. JMIR Ment Health. Apr 29, 2021;8(4):e25847. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Leech T, Dorstyn D, Taylor A, Li W. Mental health apps for adolescents and young adults: a systematic review of randomised controlled trials. Child Youth Serv Rev. Aug 2021;127:106073. [ CrossRef ]
• Buttazzoni A, Brar K, Minaker L. Smartphone-based interventions and internalizing disorders in youth: systematic review and meta-analysis. J Med Internet Res. Jan 11, 2021;23(1):e16490. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Eisenstadt M, Liverpool S, Infanti E, Ciuvat RM, Carlsson C. Mobile apps that promote emotion regulation, positive mental health, and well-being in the general population: systematic review and meta-analysis. JMIR Ment Health. Nov 08, 2021;8(11):e31170. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lipschitz J, Hogan TP, Bauer MS, Mohr DC. Closing the research-to-practice gap in digital psychiatry: the need to integrate implementation science. J Clin Psychiatry. May 14, 2019;80(3):18com12659. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hollis C. Youth mental health: risks and opportunities in the digital world. World Psychiatry. Feb 11, 2022;21(1):81-82. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bear HA, Ayala Nunes L, DeJesus J, Liverpool S, Moltrecht B, Neelakantan L, et al. Determination of markers of successful implementation of mental health apps for young people: systematic review. J Med Internet Res. Nov 09, 2022;24(11):e40347. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. Mar 19, 2011;38(2):65-76. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ramos G, Ponting C, Labao JP, Sobowale K. Considerations of diversity, equity, and inclusion in mental health apps: a scoping review of evaluation frameworks. Behav Res Ther. Dec 2021;147:103990. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ramos G, Chavira DA. Use of technology to provide mental health care for racial and ethnic minorities: evidence, promise, and challenges. Cognit Behav Pract. Feb 2022;29(1):15-40. [ CrossRef ]
• Wies B, Landers C, Ienca M. Digital mental health for young people: a scoping review of ethical promises and challenges. Front Digit Health. Sep 6, 2021;3:697072. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Liverpool S, Mota CP, Sales CM, Čuš A, Carletto S, Hancheva C, et al. Engaging children and young people in digital mental health interventions: systematic review of modes of delivery, facilitators, and barriers. J Med Internet Res. Jun 23, 2020;22(6):e16317. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Marcano Belisario JS, Huckvale K, Greenfield G, Car J, Gunn LH. Smartphone and tablet self management apps for asthma. Cochrane Database Syst Rev. Nov 27, 2013;2013(11):CD010013. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Marcano Belisario JS, Jamsek J, Huckvale K, O'Donoghue J, Morrison CP, Car J. Comparison of self-administered survey questionnaire responses collected using mobile apps versus other methods. Cochrane Database Syst Rev. Jul 27, 2015;2015(7):MR000042. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. Mar 29, 2021;372:n71. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo MP, Dagenais P, et al. The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Educ Inf. Dec 18, 2018;34(4):285-291. [ CrossRef ]
• Crabtree BF, Miller WL. A template approach to text analysis: developing and using codebooks. In: Crabtree BF, Miller WL, editors. Doing Qualitative Research. Thousand Oaks, CA. SAGE Publications, Inc; 1992:93-109.
• Newbold A, Warren FC, Taylor RS, Hulme C, Burnett S, Aas B, et al. Promotion of mental health in young adults via mobile phone app: study protocol of the ECoWeB (emotional competence for well-being in young adults) cohort multiple randomised trials. BMC Psychiatry. Sep 22, 2020;20(1):458. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. Jan 2006;3(2):77-101. [ CrossRef ]
• Ellaway RH, Kehoe A, Illing J. Critical realism and realist inquiry in medical education. Acad Med. Jul 2020;95(7):984-988. [ CrossRef ] [ Medline ]
• Willig C. Perspectives on the epistemological bases for qualitative research. In: Cooper H, Camic PM, Long DL, Panter AT, Rindskopf D, Sher KJ, editors. APA Handbook of Research Methods in Psychology, Vol. 1. Foundations, Planning, Measures, and Psychometrics. Washington, DC. American Psychological Association; 2012.
• Fletcher AJ. Applying critical realism in qualitative research: methodology meets method. Int J Soc Res Methodol. Feb 29, 2016;20(2):181-194. [ CrossRef ]
• Bohleber L, Crameri A, Eich-Stierli B, Telesko R, von Wyl A. Can we foster a culture of peer support and promote mental health in adolescence using a web-based app? A control group study. JMIR Ment Health. Sep 23, 2016;3(3):e45. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Deady M, Glozier N, Collins D, Einboden R, Lavender I, Wray A, et al. The utility of a mental health app in apprentice workers: a pilot study. Front Public Health. Sep 4, 2020;8:389. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fleming JB, Hill YN, Burns MN. Usability of a culturally informed mHealth intervention for symptoms of anxiety and depression: feedback from young sexual minority men. JMIR Hum Factors. Aug 25, 2017;4(3):e22. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Francis J, Cross D, Schultz A, Armstrong D, Nguyen R, Branch-Smith C. Developing a smartphone application to support social connectedness and wellbeing in young people with cystic fibrosis. J Cyst Fibros. Mar 2020;19(2):277-283. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Geirhos A, Domhardt M, Lunkenheimer F, Temming S, Holl RW, Minden K, et al. Feasibility and potential efficacy of a guided internet- and mobile-based CBT for adolescents and young adults with chronic medical conditions and comorbid depression or anxiety symptoms (youthCOACH): a randomized controlled pilot trial. BMC Pediatr. Jan 29, 2022;22(1):69. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Glover AC, Schueller SM, Winiarski DA, Smith DL, Karnik NS, Zalta AK. Automated mobile phone-based mental health resource for homeless youth: pilot study assessing feasibility and acceptability. JMIR Ment Health. Oct 11, 2019;6(10):e15144. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gonsalves PP, Hodgson ES, Kumar A, Aurora T, Chandak Y, Sharma R, et al. Design and development of the "POD adventures" smartphone game: a blended problem-solving intervention for adolescent mental health in India. Front Public Health. Aug 23, 2019;7:238. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gonsalves PP, Hodgson ES, Bhat B, Sharma R, Jambhale A, Michelson D, et al. App-based guided problem-solving intervention for adolescent mental health: a pilot cohort study in Indian schools. Evid Based Ment Health. Feb 18, 2021;24(1):11-18. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Haug S, Paz Castro R, Meyer C, Filler A, Kowatsch T, Schaub MP. A mobile phone-based life skills training program for substance use prevention among adolescents: pre-post study on the acceptance and potential effectiveness of the program, Ready4life. JMIR Mhealth Uhealth. Oct 04, 2017;5(10):e143. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Leonard NR, Casarjian B, Fletcher RR, Praia C, Sherpa D, Kelemen A, et al. Theoretically-based emotion regulation strategies using a mobile app and wearable sensor among homeless adolescent mothers: acceptability and feasibility study. JMIR Pediatr Parent. Mar 01, 2018;1(1):e1. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schueller SM, Glover AC, Rufa AK, Dowdle CL, Gross GD, Karnik NS, et al. A mobile phone-based intervention to improve mental health among homeless young adults: pilot feasibility trial. JMIR Mhealth Uhealth. Jul 02, 2019;7(7):e12347. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Escobar-Viera CG, Porta G, Coulter RW, Martina J, Goldbach J, Rollman BL. A chatbot-delivered intervention for optimizing social media use and reducing perceived isolation among rural-living LGBTQ+ youth: development, acceptability, usability, satisfaction, and utility. Internet Interv. Dec 2023;34:100668. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Torok M, Han J, McGillivray L, Wong Q, Werner-Seidler A, O'Dea B, et al. The effect of a therapeutic smartphone application on suicidal ideation in young adults: findings from a randomized controlled trial in Australia. PLoS Med. May 2022;19(5):e1003978. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yang YJ, Chung KM. Pilot randomized control trial of an app-based CBT program for reducing anxiety in individuals with ASD without intellectual disability. J Autism Dev Disord. Apr 2023;53(4):1331-1346. [ CrossRef ] [ Medline ]
• Children and parents: media use and attitudes report 2018. Ofcom. Mar 5, 2019. URL: https:/​/www.​ofcom.org.uk/​media-use-and-attitudes/​media-habits-children/​children-and-parents-media-use-and-attitudes-report-2018/​ [accessed 2024-07-12]
• Silver L. Smartphone ownership is growing rapidly around the world, but not always equally. Pew Research Center. Feb 5, 2019. URL: https:/​/www.​pewresearch.org/​global/​2019/​02/​05/​smartphone-ownership-is-growing-rapidly-around-the-world-but-not-always-equally/​ [accessed 2024-07-12]
• Chan A, Kow R, Cheng JK. Adolescents’ perceptions on smartphone applications (apps) for health management. JournalMTM. Aug 2017;6(2):47-55. [ CrossRef ]
• Lipschitz JM, Van Boxtel R, Torous J, Firth J, Lebovitz JG, Burdick KE, et al. Digital mental health interventions for depression: scoping review of user engagement. J Med Internet Res. Oct 14, 2022;24(10):e39204. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ng MM, Firth J, Minen M, Torous J. User engagement in mental health apps: a review of measurement, reporting, and validity. Psychiatr Serv. Jul 01, 2019;70(7):538-544. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Oakley-Girvan I, Yunis R, Longmire M, Ouillon JS. What works best to engage participants in mobile app interventions and e-health: a scoping review. Telemed J E Health. Jun 2022;28(6):768-780. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brian RM, Ben-Zeev D. Mobile health (mHealth) for mental health in Asia: objectives, strategies, and limitations. Asian J Psychiatr. Aug 2014;10:96-100. [ CrossRef ] [ Medline ]
• Li H, Lewis C, Chi H, Singleton G, Williams N. Mobile health applications for mental illnesses: an Asian context. Asian J Psychiatr. Dec 2020;54:102209. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Spielmans GI, Gatlin ET, McFall JP. The efficacy of evidence-based psychotherapies versus usual care for youths: controlling confounds in a meta-reanalysis. Psychother Res. Mar 23, 2010;20(2):234-246. [ CrossRef ] [ Medline ]
• Rapport F, Smith J, Hutchinson K, Clay-Williams R, Churruca K, Bierbaum M, et al. Too much theory and not enough practice? The challenge of implementation science application in healthcare practice. J Eval Clin Pract. Dec 15, 2022;28(6):991-1002. [ CrossRef ] [ Medline ]
• Jensen SA, Corralejo SM. Measurement issues: large effect sizes do not mean most people get better - clinical significance and the importance of individual results. Child Adolesc Ment Health. Sep 2017;22(3):163-166. [ CrossRef ] [ Medline ]
• Increasing diversity in research participation: a good practice guide for engaging with underrepresented groups. National Health Service England. Feb 14, 2023. URL: https://www.england.nhs.uk/aac/publication/increasing-diversity-in-research-participation/ [accessed 2023-03-16]
• Bergen N, Labonté R. "Everything is perfect, and we have no problems": detecting and limiting social desirability bias in qualitative research. Qual Health Res. Apr 2020;30(5):783-792. [ CrossRef ] [ Medline ]
• Vogt WP, Johnson RB. Dictionary of Statistics & Methodology: A Nontechnical Guide for the Social Sciences. Thousand Oaks, CA. SAGE Publications; 2011.
• Midgley N, Parkinson S, Holmes J, Stapley E, Eatough V, Target M. "Did I bring it on myself?" An exploratory study of the beliefs that adolescents referred to mental health services have about the causes of their depression. Eur Child Adolesc Psychiatry. Jan 20, 2017;26(1):25-34. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Willis A, Isaacs T, Khunti K. Improving diversity in research and trial participation: the challenges of language. Lancet Public Health. Jul 2021;6(7):e445-e446. [ CrossRef ]
• Ellard-Gray A, Jeffrey NK, Choubak M, Crann SE. Finding the hidden participant: solutions for recruiting hidden, hard-to-reach, and vulnerable populations. Int J Qual Method. Dec 17, 2015;14(5). [ CrossRef ]
• Moltrecht B, Patalay P, Bear HA, Deighton J, Edbrooke-Childs J. A transdiagnostic, emotion regulation app (Eda) for children: design, development, and lessons learned. JMIR Form Res. Jan 19, 2022;6(1):e28300. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bernal G, Jiménez-Chafey MI, Domenech Rodríguez MM. Cultural adaptation of treatments: a resource for considering culture in evidence-based practice. Prof Psychol Res Pract. Aug 2009;40(4):361-368. [ CrossRef ]
• Benish SG, Quintana S, Wampold BE. Culturally adapted psychotherapy and the legitimacy of myth: a direct-comparison meta-analysis. J Couns Psychol. Jul 2011;58(3):279-289. [ CrossRef ] [ Medline ]
• Hall GC, Ibaraki AY, Huang ER, Marti CN, Stice E. A meta-analysis of cultural adaptations of psychological interventions. Behav Ther. Nov 2016;47(6):993-1014. [ CrossRef ] [ Medline ]
• Huey SJJ, Polo AJ. Evidence-based psychosocial treatments for ethnic minority youth. J Clin Child Adolesc Psychol. Jan 03, 2008;37(1):262-301. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ellis DM, Draheim AA, Anderson PL. Culturally adapted digital mental health interventions for ethnic/racial minorities: a systematic review and meta-analysis. J Consult Clin Psychol. Oct 2022;90(10):717-733. [ CrossRef ] [ Medline ]
• Harper Shehadeh M, Heim E, Chowdhary N, Maercker A, Albanese E. Cultural adaptation of minimally guided interventions for common mental disorders: a systematic review and meta-analysis. JMIR Ment Health. Sep 26, 2016;3(3):e44. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Balci S, Spanhel K, Sander LB, Baumeister H. Culturally adapting internet- and mobile-based health promotion interventions might not be worth the effort: a systematic review and meta-analysis. NPJ Digit Med. Mar 23, 2022;5(1):34. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Spanhel K, Balci S, Feldhahn F, Bengel J, Baumeister H, Sander LB. Cultural adaptation of internet- and mobile-based interventions for mental disorders: a systematic review. NPJ Digit Med. Aug 25, 2021;4(1):128. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Grist R, Croker A, Denne M, Stallard P. Technology delivered interventions for depression and anxiety in children and adolescents: a systematic review and meta-analysis. Clin Child Fam Psychol Rev. Jun 18, 2019;22(2):147-171. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by T de Azevedo Cardoso, S Ma; submitted 13.05.23; peer-reviewed by P Whelan, I Vainieri, H Bao; comments to author 13.09.23; revised version received 26.09.23; accepted 10.06.24; published 30.07.24.

©Holly Alice Bear, Lara Ayala Nunes, Giovanni Ramos, Tanya Manchanda, Blossom Fernandes, Sophia Chabursky, Sabine Walper, Edward Watkins, Mina Fazel. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 30.07.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Qualitative systematic reviews: their importance for our understanding of research relevant to pain

Affiliation.

• 1 RCN Research Institute, Warwick Medical School, University of Warwick, Coventry, UK.
• PMID: 26516555
• PMCID: PMC4616987
• DOI: 10.1177/2049463714549777

This article outlines what a qualitative systematic review is and explores what it can contribute to our understanding of pain. Many of us use evidence of effectiveness for various interventions when working with people in pain. A good systematic review can be invaluable in bringing together research evidence to help inform our practice and help us understand what works. In addition to evidence of effectiveness, understanding how people with pain experience both their pain and their care can help us when we are working with them to provide care that meets their needs. A rigorous qualitative systematic review can also uncover new understandings, often helping illuminate 'why' and can help build theory. Such a review can answer the question 'What is it like to have chronic pain?' This article presents the different stages of meta-ethnography, which is the most common methodology used for qualitative systematic reviews. It presents evidence from four meta-ethnographies relevant to pain to illustrate the types of findings that can emerge from this approach. It shows how new understandings may emerge and gives an example of chronic musculoskeletal pain being experienced as 'an adversarial struggle' across many aspects of the person's life. This article concludes that evidence from qualitative systematic reviews has its place alongside or integrated with evidence from more quantitative approaches.

Keywords: Qualitative systematic review; meta-ethnography; qualitative synthesis.

PubMed Disclaimer

Conflict of interest statement

Declaration of Conflicting Interests: The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Similar articles

• A meta-ethnography of patients’ experience of chronic non-malignant musculoskeletal pain. Toye F, Seers K, Allcock N, Briggs M, Carr E, Andrews J, Barker K. Toye F, et al. Southampton (UK): NIHR Journals Library; 2013 Nov. Southampton (UK): NIHR Journals Library; 2013 Nov. PMID: 25642538 Free Books & Documents. Review.
• Student and educator experiences of maternal-child simulation-based learning: a systematic review of qualitative evidence protocol. MacKinnon K, Marcellus L, Rivers J, Gordon C, Ryan M, Butcher D. MacKinnon K, et al. JBI Database System Rev Implement Rep. 2015 Jan;13(1):14-26. doi: 10.11124/jbisrir-2015-1694. JBI Database System Rev Implement Rep. 2015. PMID: 26447004
• The Effectiveness of Integrated Care Pathways for Adults and Children in Health Care Settings: A Systematic Review. Allen D, Gillen E, Rixson L. Allen D, et al. JBI Libr Syst Rev. 2009;7(3):80-129. doi: 10.11124/01938924-200907030-00001. JBI Libr Syst Rev. 2009. PMID: 27820426
• Promoting and supporting self-management for adults living in the community with physical chronic illness: A systematic review of the effectiveness and meaningfulness of the patient-practitioner encounter. Rees S, Williams A. Rees S, et al. JBI Libr Syst Rev. 2009;7(13):492-582. doi: 10.11124/01938924-200907130-00001. JBI Libr Syst Rev. 2009. PMID: 27819974
• Health professionals' experience of teamwork education in acute hospital settings: a systematic review of qualitative literature. Eddy K, Jordan Z, Stephenson M. Eddy K, et al. JBI Database System Rev Implement Rep. 2016 Apr;14(4):96-137. doi: 10.11124/JBISRIR-2016-1843. JBI Database System Rev Implement Rep. 2016. PMID: 27532314 Review.
• Telemedicine in Advanced Kidney Disease and Kidney Transplant: A Qualitative Meta-Analysis of Studies of Patient Perspectives. Manko CD, Apple BJ, Chang AR, Romagnoli KM, Johannes BL. Manko CD, et al. Kidney Med. 2024 May 24;6(7):100849. doi: 10.1016/j.xkme.2024.100849. eCollection 2024 Jul. Kidney Med. 2024. PMID: 39040545 Free PMC article.
• What are the strengths and limitations to utilising creative methods in public and patient involvement in health and social care research? A qualitative systematic review. Phillips OR, Harries C, Leonardi-Bee J, Knight H, Sherar LB, Varela-Mato V, Morling JR. Phillips OR, et al. Res Involv Engagem. 2024 May 13;10(1):48. doi: 10.1186/s40900-024-00580-4. Res Involv Engagem. 2024. PMID: 38741156 Free PMC article. Review.
• Optometry student clinical practice at public health facilities: A systematic review. Segooa RF, Moodley VR. Segooa RF, et al. Health SA. 2024 Mar 22;29:2441. doi: 10.4102/hsag.v29i0.2441. eCollection 2024. Health SA. 2024. PMID: 38628230 Free PMC article.
• Drivers of inappropriate use of antimicrobials in South Asia: A systematic review of qualitative literature. Murray JL, Leung DT, Hanson OR, Ahmed SM, Pavia AT, Khan AI, Szymczak JE, Vaughn VM, Patel PK, Biswas D, Watt MH. Murray JL, et al. PLOS Glob Public Health. 2024 Apr 4;4(4):e0002507. doi: 10.1371/journal.pgph.0002507. eCollection 2024. PLOS Glob Public Health. 2024. PMID: 38573955 Free PMC article.
• Socio-Ecological Analysis of Barriers to Access and Utilization of Adolescent Sexual and Reproductive Health Services in Sub-Saharan Africa: A Qualitative Systematic Review. Sidamo NB, Kerbo AA, Gidebo KD, Wado YD. Sidamo NB, et al. Open Access J Contracept. 2023 Jun 26;14:103-118. doi: 10.2147/OAJC.S411924. eCollection 2023. Open Access J Contracept. 2023. PMID: 37398897 Free PMC article. Review.
• Campbell R, Pound P, Morgan M, et al. Evaluating meta-ethnography: systematic analysis and synthesis of qualitative research. Health Technol Assess 2011; 15: 1–164. - PubMed
• Toye F, Seers K, Allcock N, et al. A meta-ethnography of patients’ experience of chronic non-malignant musculoskeletal pain. Health Serv Deliv Res 2013; 1(12). - PubMed
• Gülmezoglu AM, Chandler J, Shepperd S, et al. Reviews of qualitative evidence: a new milestone for Cochrane (editorial). Cochrane Database Syst Rev 2013; 11: ED000073, dx.doi.org/10.1002/14651858.ED000073 - DOI - PMC - PubMed
• Seers K. What is a qualitative synthesis? Evid Base Nurs 2012; 15(4): 101. - PubMed
• Noblit G, Hare R. Meta-ethnography: synthesising qualitative studies (Qualitative research methods, Series 11). Thousand Oaks, CA: SAGE, 1988.

Related information

• Cited in Books

LinkOut - more resources

Full text sources.

• Europe PubMed Central
• Ovid Technologies, Inc.
• PubMed Central

Other Literature Sources

• scite Smart Citations
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Managing neurodiversity in workplaces: a review and future research agenda for sustainable human resource management.

1. Introduction

2. literature review, 2.1. neurodiversity—origins of the concept and statistics, 2.2. neurodiversity in the labour market, 2.3. managing neurodiversity in workplaces—organizational barriers and needs, 3. materials and methods.

• What are the research directions and gaps in the topic of managing neurodiversity in workplaces mentioned in the literature?
• What are future research directions in the field of managing neurodiversity in workplaces with reference to Methodology, Theory, and Contexts?

5. Discussion and Conclusions

Author contributions, data availability statement, conflicts of interest.

• Honeybourne, V. The Neurodiverse Workplace: An Employer’s Guide to Managing and Working with Neurodivergent Employees, Clients and Customers ; Jessica Kingsley Publishers: London, UK, 2019. [ Google Scholar ]
• Hyland, S.; Connolly, J. Companies Leading in Disability Inclusion Have Outperformed Peers. Accenture Research Finds. Available online: https://newsroom.accenture.com/news/2018/companies-leading-in-disability-inclusion-have-outperformed-peers-accenture-research-finds (accessed on 4 July 2024).
• Jefferies, J.G.; Ahmed, W. Marketing Neurodiversity for Well-Being. J. Consum. Mark. 2022 , 39 , 632–648. [ Google Scholar ] [ CrossRef ]
• Whelpley, C.E.; Perrault, E. Autism at Work: How Internal and External Factors Influence Employee Outcomes and Firm Performance. J. Gen. Manag. 2021 , 46 , 210–219. [ Google Scholar ] [ CrossRef ]
• Vogus, T.J.; Taylor, J.L. Flipping the Script: Bringing an Organizational Perspective to the Study of Autism at Work. Autism 2018 , 22 , 514–516. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Campos-García, I.; Alonso-Muñoz, S.; González-Sánchez, R.; Medina-Salgado, M.-S. Human Resource Management and Sustainability: Bridging the 2030 Agenda. Corp. Soc. Responsib. Environ. Manag. 2024 , 31 , 2033–2053. [ Google Scholar ] [ CrossRef ]
• Ehnert, I.; Harry, W.; Zink, K.J. (Eds.) Sustainability and Human Resource Management. Developing Sustainable Business Organizations ; Springer: Berlin/Heidelberg, Germany, 2014. [ Google Scholar ] [ CrossRef ]
• Sarkis, J.; Sundaram, A. An Examination of Sustainable Human Resource Management Practices in Developing Countries. J. Clean. Prod. 2016 , 114 , 664–673. [ Google Scholar ]
• Flamini, G.; Gnan, L. Sustainable HRM. In Organizing Sustainable Development ; Routledge: London, UK, 2023; pp. 172–186. [ Google Scholar ] [ CrossRef ]
• Singer, J. Odd People in: The Birth of Community Amongst People on the Autism Spectrum: A Personal Exploration of a New Social Movement Based on Neurological Diversity ; University of Technology: Sydney, Australia, 1998. [ Google Scholar ]
• Armstrong, T. The Power of Neurodiversity: Unleashing the Advantages of Your Differently Wired Brain ; Da Capo Press: Philadelphia, PA, USA, 2011. [ Google Scholar ]
• Rothstein, A. Review: Mental Disorder or Neurodiversity? New Atlantis 2012 , 36 , 99–115. [ Google Scholar ]
• Singer, J. Why Can’t You Be Normal for Once in Your Life? From a Problem with No Name to the Emergence of a New Category of Difference. In Disability Discourse ; Corker, M., French, S., Eds.; Open University Press: Buckingham, UK, 1999; pp. 59–67. [ Google Scholar ]
• Hendrickx, S. The Adolescent and Adult Neuro-Diversity Handbook: Asperger’s Syndrome, ADHD, Dyslexia, Dyspraxia, and Related Conditions ; Jessica Kingsley Publishers: London, UK, 2010. [ Google Scholar ]
• McGee, M. Neurodiversity. Contexts 2012 , 11 , 12–13. [ Google Scholar ] [ CrossRef ]
• Maenner, M.J.; Shaw, K.A.; Baio, J.; Washington, A.; Patrick, M.; DiRienzo, K.; Christensen, D.L.; Wiggins, D.L.; Pettygrove, S.; Andrews, J.G.; et al. Prevalence of Autism Spectrum Disorder Among Children Aged 8 Years—Autism and Developmental Disabilities Monitoring Network, 11 Sites, United States, 2016. CDC MMWR Surveill. Summ. 2020 , 69 , 1–16. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Polskiautyzm.pl. Available online: https://polskiautyzm.pl/ (accessed on 28 June 2024).
• Supreme Audit Office. Available online: https://www.nik.gov.pl/plik/id,23545.pdf (accessed on 28 June 2024).
• Wiater, M. Innowacje Społeczne w Obszarze Neuroróżnorodności. Eduk. Ekon. i Menedżerów 2022 , 64 , 37–49. Available online: https://econjournals.sgh.waw.pl/EEiM/article/view/2989 (accessed on 28 June 2024). [ CrossRef ]
• Whelpley, C.E.; Banks, G.C.; Bochantin, J.E.; Sandoval, R. Tensions on the Spectrum: An Inductive Investigation of Employee and Manager Experiences of Autism. J. Bus. Psychol. 2020 , 36 , 283–297. [ Google Scholar ] [ CrossRef ]
• Morath, E. America’s Hidden Workforce Returns; Disabled Workers Join Job Market at Highest Level in Years Amid Low Unemployment and Stricter Rules for Claiming Benefits. Wall Street Journal. Available online: https://www.wsj.com/articles/americas-hidden-workforce-returns-11548478801 (accessed on 28 June 2024).
• American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders , 5th ed.; American Psychiatric Publishing: Arlington, VA, USA, 2013. [ Google Scholar ]
• Smith, M.D.; Belcher, R.G.; Juhrs, P.D. A Guide to Successful Employment for Individuals with Autism ; Brookes Publishing Company: Baltimore, MD, USA, 1997. [ Google Scholar ]
• Austin, R.D.; Pisano, G.P. Neurodiversity as a Competitive Advantage. Harv. Bus. Rev. 2017 , 95 , 96–103. [ Google Scholar ]
• Howlin, P.; Goode, S.; Hutton, J.; Rutter, M. Adult Outcome for Children with Autism. J. Child Psychol. Psychiatry 2004 , 45 , 212–229. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Krumm, N.; Turner, T.N.; Baker, C.; Vives, L.; Mohajeri, K.; Witherspoon, K.; Raja, A.; Coe, B.P.; Stessman, H.A.; He, Z.X.; et al. Excess of Rare, Inherited Truncating Mutations in Autism. Nat. Genet. 2015 , 47 , 582–588. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gerhardt, P.F.; Lainer, I. Addressing the Needs of Adolescents and Adults with Autism: A Crisis on the Horizon. J. Contemp. Psychother. 2011 , 41 , 37–45. [ Google Scholar ] [ CrossRef ]
• Polish Economic Institute; JiM Foundation. Raport Praca Autyzm Jim Pik ; Polish Economic Institute: Warsaw, Poland, 2022; p. 4. Available online: https://jim.org/en/raport_praca_autyzm_jim_pik/ (accessed on 28 June 2024).
• Wiater, M. Wyzwania Związane z Zatrudnianiem Osób Neuroróżnorodnych w Polsce. HRM Zarz. Zasob. Ludzk. 2023 , 154 , 80–92. [ Google Scholar ]
• Fundacja Synapsis. Available online: https://synapsis.org.pl/ (accessed on 28 June 2024).
• Johnson, T.D.; Joshi, A. Dark Clouds or Silver Linings? A Stigma Threat Perspective on the Implications of an Autism Diagnosis for Workplace Well-Being. J. Appl. Psychol. 2016 , 101 , 430–449. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Lorenz, T.; Frischling, C.; Cuadros, R.; Heinitz, K. Autism and Overcoming Job Barriers: Comparing Job-Related Barriers and Possible Solutions In and Outside of Autism-Specific Employment. PLoS ONE 2016 , 11 , e0147040. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Neely, B.H.; Hunter, S.T. In a Discussion of Invisible Disabilities, Let Us not Lose Sight of Employees on the Autism Spectrum. Ind. Organ. Psychol. 2014 , 7 , 274–277. [ Google Scholar ] [ CrossRef ]
• Hill, E.L. Linking Clinical and Industrial Psychology: Autism Spectrum Disorder at Work. Ind. Organ. Psychol. 2014 , 7 , 152–155. [ Google Scholar ] [ CrossRef ]
• Richards, J.; Sang, K.; Marks, A.; Gill, S. “I’ve Found It Extremely Draining” Emotional Labour and the Lived Experience of Line Managing Neurodiversity. Pers. Rev. 2019 , 48 , 1903–1923. [ Google Scholar ] [ CrossRef ]
• Ryan, R.M.; Deci, E.L. Self-Determination Theory and the Facilitation of Intrinsic Motivation, Social Development, and Well-Being. Am. Psychol. 2000 , 55 , 68–78. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Berwick, K.R. Stress Among Student Affairs Administrators: The Relationship of Personal Characteristics and Organizational Variables to Work-Related Stress. J. Coll. Stud. Dev. 1992 , 33 , 11–19. [ Google Scholar ]
• Conley, V.M. Separation: An Internal Aspect of the Staffing Process. Coll. Stud. Aff. J. 2001 , 21 , 57–63. [ Google Scholar ]
• Tull, A. Synergistic Supervision, Job Satisfaction, and Intention to Turnover of New Professionals in Student Affairs. J. Coll. Stud. Dev. 2006 , 4 , 465–480. [ Google Scholar ] [ CrossRef ]
• Snyder, H. Literature Review as a Research Methodology: An Overview and Guidelines. J. Bus. Res. 2019 , 104 , 333–339. [ Google Scholar ] [ CrossRef ]
• Paul, J.; Criado, A.R. The Art of Writing Literature Review: What Do We Know and What Do We Need to Know? Int. Bus. Rev. 2020 , 29 , 1–7. [ Google Scholar ] [ CrossRef ]
• Rollnik-Sadowska, E. Labor Market in Sustainability Transitions: A Systematic Literature Review. Econ. Environ. 2023 , 87 , 4. [ Google Scholar ] [ CrossRef ]
• Ejdys, J.; Szpilko, D. European Green Deal–Research Directions. Syst. Lit. Review. Econ. Environ. 2022 , 81 , 8–38. [ Google Scholar ] [ CrossRef ]
• Barkun, Y.; Rollnik-Sadowska, E.; Glińska, E. The Concept of ‘Talent’ in the Labor Management Perspective-The Bibliometric Analysis of Literature. Int. J. Ind. Eng. Manag. 2020 , 12 , 104–115. [ Google Scholar ] [ CrossRef ]
• Kaššaj, M.; Peráček, T. Synergies and potential of Industry 4.0 and automated vehicles in smart city infrastructure. Appl. Sci. 2024 , 14 , 3575. [ Google Scholar ] [ CrossRef ]
• Krzeminska, A.; Austin, R.; Bruyére, S.; Hedley, D. The Advantages and Challenges of Neurodiversity Employment in Organizations. J. Manag. Organ. 2019 , 25 , 453–463. [ Google Scholar ] [ CrossRef ]
• Morrison, K.E.; DeBrabander, K.M.; Jones, D.R.; Ackerman, R.A.; Sasson, N.J. Social Cognition, Social Skill, and Social Motivation Minimally Predict Social Interaction Outcomes for Autistic and Non-Autistic Adults. Front. Psychol. 2020 , 11 , 1–22. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Stenn, T.; Lalor, A.R.; Coplan, J.; Osterholt, D.A. Managing neurodiversity inclusion in today’s entrepreneurial-styled workplace. In Managing for Social Justice. Harnessing Management Theory and Practice for Collective Good ; Poonamalle, L., Howard, A.D., Joy, S., Eds.; Springer International Publishing: Cham, Switzerland, 2023. [ Google Scholar ]
• Doyle, N.; McDowall, A. Diamond in the Rough? An ‘Empty Review’ of Research into ‘Neurodiversity’ and a Road Map for Developing the Inclusion Agenda. Equal. Divers. Incl. 2022 , 41 , 352–382. [ Google Scholar ] [ CrossRef ]
• Muhammad, R.; Omkar, D.; Duan, X. Leadership Strategies for Effective Diversity Management ; IGI Global: Hershey, PA, USA, 2023. [ Google Scholar ] [ CrossRef ]
• Khan, M.H.; Grabarski, M.K.; Ali, M.; Buckmaster, S. Insights into Creating and Managing an Inclusive Neurodiverse Workplace for Positive Outcomes: A Multistaged Theoretical Framework. Group Organ. Manag. 2023 , 48 , 1339–1386. [ Google Scholar ] [ CrossRef ]
• Szulc, J.M.; McGregor, F.-L.; Cakir, E. Neurodiversity and Remote Work in Times of Crisis: Lessons for HR. Pers. Rev. 2023 , 52 , 1677–1692. [ Google Scholar ] [ CrossRef ]
• Howes, S.S. Emotional Intelligence in Autistic Adults: A Review with Considerations for Employers. Sustainability 2023 , 15 , 7252. [ Google Scholar ] [ CrossRef ]
• Doyle, N.; Bradley, E. Disability Coaching in a Pandemic. J. Work. -Appl. Manag. 2023 , 15 , 135–147. [ Google Scholar ] [ CrossRef ]
• Doyle, N.; McDowall, A. Neurodiversity Coaching: A Psychological Approach to Supporting Neurodivergent Talent and Career Potential , 1st ed.; Routledge: London, UK, 2023. [ Google Scholar ] [ CrossRef ]
• Allen, R.A.; Dickmann, M.; Priscott, T.; White, G.R.T. Exploring Positive and Negative Intersectionality Effects: An Employment Study of Neurodiverse UK Military Veterans. Int. J. Hum. Resour. Manag. 2023 . [ Google Scholar ] [ CrossRef ]
• Crook, T.; McDowall, A. Paradoxical Career Strengths and Successes of ADHD Adults: An Evolving Narrative. J. Work. Manag. 2023 , 16 , 112–126. [ Google Scholar ] [ CrossRef ]
• Waterbury, C.J.; Smith, N. Invisible Diversity in the Hospitality Industry. In The Routledge Handbook of Diversity, Equity, and Inclusion Management in the Hospitality Industry ; Routledge: London, UK, 2023; pp. 128–139. [ Google Scholar ]
• Branton, S.E.; Villamil, A.M.; Reed, J.L. Branding Neurodiversity: A Critical Discourse Analysis of Communicative Capitalism and Change Empowerment Among Neurodiversity Workforce Intermediaries. J. Public Relat. Res. 2023 , 35 , 357–374. [ Google Scholar ] [ CrossRef ]
• Warnock, R. Relational Legacies and Relative Experiences: Austerity, Inequality, and Access to Special Educational Needs and Disability (SEND) Support in London, England. Geogr. J. 2023 , 1–12. [ Google Scholar ] [ CrossRef ]
• Whelpley, C.E.; Woznyj, H.M. Balancing the Teeter Totter: A Dialectical View of Managing Neurodiverse Employees. Hum. Perform. 2023 , 36 , 133–154. [ Google Scholar ] [ CrossRef ]
• Szulc, J.M. Towards More Inclusive Qualitative Research: The Practice of Interviewing Neurominorities. Labour Ind. 2023 , 33 , 179–187. [ Google Scholar ] [ CrossRef ]
• Turowetz, J.; Wiscons, L.Z.; Maynard, D.W. Disorder or Difference? How Clinician-Patient Interaction and Patient Age Shape the Process and Meaning of Autism Diagnosis. Sociol. Heal. Illn. 2023 , 46 , 171–188. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Tomczak, M.T. How Can the Work Environment Be Redesigned to Enhance the Well-Being of Individuals with Autism? Empl. Relat. 2022 , 44 , 1467–1484. [ Google Scholar ] [ CrossRef ]
• Spoor, J.R.; Flower, R.L.; Bury, S.M.; Hedley, D. Employee Engagement and Commitment to Two Australian Autism Employment Programs: Associations with Workload and Perceived Supervisor Support. Equal. Divers. Incl. 2022 , 41 , 508–524. [ Google Scholar ] [ CrossRef ]
• Hinder, A.M. Stellar Evolution: Career Paths of Autistic Professionals in Corporate Law Firms. In Emerald Studies in Workplace Neurodiversity ; Generation, A., Hurley-Hanson, A.E., Giannantonio, C.M., Eds.; Emerald Publishing Limited: Leeds, UK, 2022; pp. 65–82. [ Google Scholar ] [ CrossRef ]
• Molloy, A.; O‘Donoghue, A.; Fu, N. Enabling Neurodiversity in the Workplace via Inclusive Human Resource Practices. In Emerald Studies in Workplace Neurodiversity ; Generation, A., Hurley-Hanson, A.E., Giannantonio, C.M., Eds.; Emerald Publishing Limited: Leeds, UK, 2022; pp. 85–109. [ Google Scholar ] [ CrossRef ]
• Karol, E.; Smith, D. Design Considerations for Residents with Impeded Cognitive Functioning: Conversations with People with Schizophrenia. Sustainability 2021 , 13 , 7733. [ Google Scholar ] [ CrossRef ]
• Walkowiak, E. Neurodiversity of the Workforce and Digital Transformation: The Case of Inclusion of Autistic Workers at the Workplace. Technol. Forecast. Soc. Chang. 2021 , 168 , 120739. [ Google Scholar ] [ CrossRef ]
• Roberson, Q.; Quigley, N.R.; Vickers, K.; Bruck, I. Reconceptualizing Leadership from a Neurodiverse Perspective. Group Organ. Manag. 2021 , 46 , 399–423. [ Google Scholar ] [ CrossRef ]
• Oseland, N. Beyond the Workplace Zoo: Humanising the Office , 1st ed.; Routledge: London, UK, 2021. [ Google Scholar ] [ CrossRef ]
• Szulc, J.M.; Davies, J.; Tomczak, M.T.; McGregor, F.-L. AMO Perspectives on the Well-Being of Neurodivergent Human Capital. Empl. Relat. 2021 , 43 , 858–872. [ Google Scholar ] [ CrossRef ]
• Priscott, T.; Allen, R.A. Human Capital Neurodiversity: An Examination of Stereotype Threat Anticipation. Empl. Relat. 2021 , 43 , 1067–1082. [ Google Scholar ] [ CrossRef ]
• Saline, S. Thriving in the New Normal: How COVID-19 Has Affected Alternative Learners and Their Families and Implementing Effective, Creative Therapeutic Interventions. Smith Coll. Stud. Soc. Work. 2021 , 91 , 1–28. [ Google Scholar ] [ CrossRef ]
• Kwok, K.; Kwok, D.K. More than Comfort and Discomfort: Emotion Work of Parenting Children with Autism in Hong Kong. Child. Youth Serv. Rev. 2020 , 118 , 105456. [ Google Scholar ] [ CrossRef ]
• Aubineau, M.; Blicharska, T. High-Functioning Autistic Students Speak about Their Experience of Inclusion in Mainstream Secondary Schools. Sch. Ment. Health 2020 , 12 , 537–555. [ Google Scholar ] [ CrossRef ]
• Carrero, J.; Krzeminska, A.; Härtel, C.E.J. The DXC Technology Work Experience Program: Disability-Inclusive Recruitment and Selection in Action. J. Manag. Organ. 2019 , 25 , 535–542. [ Google Scholar ] [ CrossRef ]
• Patton, E. Autism, Attributions and Accommodations: Overcoming Barriers and Integrating a Neurodiverse Workforce. Pers. Rev. 2019 , 48 , 915–934. [ Google Scholar ] [ CrossRef ]
• Sarrett, J.C. Custodial Homes, Therapeutic Homes, and Parental Acceptance: Parental Experiences of Autism in Kerala, India, and Atlanta, GA USA. Cult. Med. Psychiatry 2015 , 39 , 254–276. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Sumner, K.E.; Brown, T.J. Neurodiversity and Human Resource Management: Employer Challenges for Applicants and Employees with Learning Disabilities. Psychol.-Manag. J. 2015 , 18 , 77–85. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Rollnik-Sadowska, E.; Grabińska, V. Managing Neurodiversity in Workplaces: A Review and Future Research Agenda for Sustainable Human Resource Management. Sustainability 2024 , 16 , 6594. https://doi.org/10.3390/su16156594

Rollnik-Sadowska E, Grabińska V. Managing Neurodiversity in Workplaces: A Review and Future Research Agenda for Sustainable Human Resource Management. Sustainability . 2024; 16(15):6594. https://doi.org/10.3390/su16156594

Rollnik-Sadowska, Ewa, and Violetta Grabińska. 2024. "Managing Neurodiversity in Workplaces: A Review and Future Research Agenda for Sustainable Human Resource Management" Sustainability 16, no. 15: 6594. https://doi.org/10.3390/su16156594

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Galdas P, Darwin Z, Fell J, et al. A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN). Southampton (UK): NIHR Journals Library; 2015 Aug. (Health Services and Delivery Research, No. 3.34.)

A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN).

Chapter 3 qualitative review methods.

The objective of the qualitative metaethnography was to systematically identify experiences of, and perceptions of, interventions or specific activities aimed at supporting or promoting self-management of LTCs among men of differing age, ethnicity and socioeconomic background.

A summary of the methods used in the metaethnography is provided in Appendix 3 , using the enhancing transparency in reporting the synthesis of qualitative research (ENTREQ) reporting standards for qualitative evidence synthesis, developed by Tong et al. 93

The evidence synthesis was conducted using a metaethnography approach originally described by Noblit and Hare. 94 This approach was chosen because of its emphasis on conceptual development and generating new insights (i.e. being interpretive rather than integrative 94 ) and because it is compatible with synthesising all types of qualitative research. 95

Metaethnography involves seven stages: getting started, deciding what is relevant, reading the studies, determining how studies are related to each other, translating studies into each other, synthesising translations and expressing the synthesis; 94 these seven, often overlapping, stages are depicted in Figure 7 .

Seven steps of metaethnography.

• Step 1: getting started

The first stage involved identifying a ‘worthy’ research question and one that could be addressed through qualitative evidence synthesis. 94 This stage took place in developing the original funding application for the current review and its justification is presented in Chapter 1 .

• Step 2: deciding what is relevant

The second stage, ‘deciding what is relevant’, was viewed as comprising the search strategy, inclusion criteria and quality appraisal, consistent with the experiences of Atkins et al. 96 These are presented next, before steps 3–7 are described in the section Data extraction strategy and data analysis .

• Search methods

Search strategy

A comprehensive electronic search strategy ( Appendix 4 ) was developed in liaison with information specialists. It sought to identify all available studies, rather than using purposive sampling to identify all available concepts. Five electronic databases were searched in July 2013 [Cumulative Index to Nursing and Allied Health Literature (CINAHL), EMBASE, MEDLINE, PsycINFO and Social Science Citation Index].

Because of challenges with methodological indexing of qualitative research, 97 the electronic search was complemented by checking reference lists, and using an adapted strategy published elsewhere 98 that includes ‘thesaurus terms’ (keywords indexed in electronic databases, e.g. ‘Qualitative Research’), ‘free text terms’ (commonly used research methodology terms searched for in the titles, abstracts and keywords) and ‘broad-based terms’ (i.e. the broad free-text terms ‘qualitative’, ‘findings’ and ‘interview$’ and the thesaurus term ‘Interviews’). Terms relating to gender were combined with other terms to narrow the search and increase the precision of the strategy (e.g. ‘men’, ‘male’, ‘masculine$’, ‘gender’, ‘sex difference$’, ‘sex factors’).

Study selection: study screening methods and inclusion criteria

Records were initially screened by one reviewer (ZD) on the basis of the title and abstract. Decisions were recorded in EndNote X7.0.2 (Thomson Reuters, CA, USA), a reference management database. All articles identified as potentially eligible for inclusion were obtained in full. Attempts were made to identify and obtain published findings for unpublished literature that was otherwise eligible, for example doctoral theses or conference proceedings.

The full-text literature was screened independently by two reviewers (ZD and PG) using the inclusion criteria listed in Table 5 . Studies that explored the experiences of men alone, or included a clear and explicit comparison between men and women, were included. Studies which focused on self-management experiences of people with LTCs more generally (i.e. did not consider experiences of, or perceptions of, a self-management support intervention or activity) were excluded. The approach to screening was inclusive; for example, studies where the qualitative findings were limited (e.g. Iredale et al. , 99 Ramachandra et al. , 100 Smith et al. 101 ) and mixed-sex studies with limited findings on gender comparisons (e.g. Barlow et al. 102 , 103 ) were retained in case they contributed to the synthesis.

Screening criteria: qualitative

• Classification of self-management interventions and support activities in the qualitative evidence synthesis

The original study protocol sought to code self-management interventions and support activities using the most up-to-date version of the taxonomy of BCT. 104 – 106 As in the quantitative review (see Chapter 2 , Coding interventions for analysis ), we found that the level of detail reported on self-management interventions or activities in the qualitative literature was limited in detail, precision and consistency, making coding with the BCT taxonomy unfeasible.

Most of the qualitative literature did not focus on behaviour change per se or seek to address men’s views and experiences of behaviour change techniques; for example, some papers were concerned with the dynamics of social support groups, or the use of other self-management support and information. The BCT taxonomy is applicable to only studies that are judged as targeting behaviour change; we were therefore limited to ‘lifestyle’ and ‘psychological’ studies. Only a minority of the studies ( n  = 13) provided sufficient information on interventions to allow even rudimentary coding with the BCT taxonomy, and these are presented in Appendix 5 . Issues around application of the BCT taxonomy are returned to in the discussion chapter (see Chapter 6 ).

The lack of detail reported in the qualitative literature also made it unfeasible to classify interventions using the system developed for the quantitative review. Whereas the quantitative review concerned trials of specific interventions, approximately half of the studies in the qualitative review 99 , 101 , 107 – 130 included more than one intervention or activity (e.g. ‘any cancer support group’).

We therefore developed a broad system for classifying interventions and support activities that offered a pragmatic way to group studies and make the analysis process more manageable. The categories are shown in Table 6 .

Categories and descriptions of self-management interventions and support activities in the qualitative evidence synthesis

• Quality assessment strategy

The purpose of quality appraisal in the review was to provide descriptive information on the quality of the included studies rather than as a basis for inclusion. We considered that studies of weaker quality either would not contribute or would contribute only minimally to the final synthesis. 94 , 131 We therefore chose not to use design-specific appraisal tools (which the original protocol stated we would) because we placed emphasis on conceptual contribution, which did not require a detailed design-specific appraisal of methodological quality. With that in mind, we used the Critical Appraisal Skills Programme (CASP) tool. 132

The CASP tool comprises 10 checklist-style questions (see Appendix 6 ) for assessing the quality of various domains (including aims, design, methods, data analysis, interpretation, findings and value of the research). Because of the checklist nature of the CASP tool, we developed some additional questions informed by other metaethnography studies 96 , 131 that enabled us to extract and record more detailed narrative summaries of the main strengths, limitations and concerns of each study (see Appendix 7 ).

The CASP tool was used in the light of the experiences reported by other researchers who recommended that, despite rather low inter-rater agreement, such an approach ‘encourag[es] the reviewers to read the papers carefully and systematically, and serves as a reminder to treat the papers as data for the synthesis’ (p. 44). 131

Its focus is on procedural aspects of the conduct of the research rather than the insights offered. 133 The quality appraisal (which focused on methodological quality) did not form part of the inclusion criteria because, as recognised by Campbell et al. , 131 it is conceptual quality that is most important for evidence synthesis and it is the process of synthesis that judges the ‘worth’ of studies, with conceptually limited studies making a limited contribution. 94 Additionally, it is acknowledged that agreement is often slight, with low reproducibility. 131 , 133 Appraisal was conducted by two reviewers independently (ZD and PG), with discrepancies resolved through discussion.

Search outcome

The electronic search strategy identified 6330 unique references. Screening based on title and abstract identified 149 papers for full-text screening. Dual screening of these full-text articles identified 34 studies (reported in 38 papers) to be included in the review. Reasons for excluding the remaining 111 articles are shown in Table 7 .

Reasons for exclusion of full-text articles

Inter-rater agreement on the decision to include was 88.6%. The majority of disagreements ( n  = 17) concerned the definition of self-management intervention or activity. Having discussed the 17 disagreements, we agreed that five studies on which there was disagreement would be included. 100 , 103 , 110 , 116 , 134

An additional four studies were identified through reference checks and efforts to locate published literature linked to unpublished work identified through the electronic search. 111 , 112 , 135 , 136 An additional two papers (women only), although individually ineligible, were located as ‘linked papers’ for two of the original 34 studies, 114 , 120 giving a total of 38 studies (reported in 44 papers), as shown in Figure 8 .

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram for the qualitative review.

• Data extraction strategy and data analysis

The lead reviewer (ZD) extracted all papers using data extraction forms previously tested and refined through a pilot study of four papers. All study details (including aim, participant details, methodology, method of data collection and analysis) were extracted into Microsoft Excel ® version 14 (Microsoft Corporation, Redmond, WA, USA) and checked by a second reviewer (PG). Extraction and analysis of study findings was undertaken by a group of coreviewers within the research team (ZD, PG, LK, CB, KM, KH) and followed steps 3–7 of the metaethnography process described by Noblit and Hare. 94 Despite being numbered sequentially, these phases do not occur in a linear process. 94

Step 3: reading the studies

The metaethnography process involved three levels of constructs, as described by Schutz 137 and operationalised by Atkins et al. : 96

• first-order: participant quotes and participant observations, while recognising that in secondary analysis these represent the participants’ views as selected by the study authors in evidencing their second-order constructs
• second-order: study authors’ themes/concepts and interpretations, also described by Noblit and Hare 94 as ‘metaphors’
• third-order: our ‘interpretations of interpretations of interpretations’ (p. 35), 94 based on our analysis of the first-order and second-order constructs extracted from the studies.

Each paper was read in full and copied verbatim into NVivo version 10 (QSR International, Warrington, UK) for line-by-line coding by the lead reviewer. Coding involved repeated reading and line-by-line categorising of first-order and second-order constructs, using participants’ and authors’ words wherever possible, and reading for possible third-order constructs.

Third-order constructs were developed by building second-order constructs into broader categories and themes in a framework which was revised iteratively using the hierarchical functions of the NVivo software (i.e. using ‘parent’ and ‘child’ nodes).

Rather than simply being a synthesis of the second-order constructs, third-order-constructs were derived inductively from the extracted data; this was an interpretive process that was not limited to interpretations offered by the original authors of included studies.

Coding by coreviewers (i.e. other members of the research team) was idiosyncratic but commonly involved working with printed papers, noting key ‘metaphors’ (themes, concepts and ideas) in the margins and highlighting first-order and second-order evidence that supported the coreviewers’ interpretations. The lead reviewer, ZD, met with each coreviewer to discuss/debrief coding decisions and ensure the credibility (i.e. the congruence of coding decisions with the original author interpretations) of the overall analytical process.

Step 4: determining how the studies are related

To offer a ‘way in’ to the synthesis, we adopted a similar approach to that of Campbell et al. : 131 initially grouping studies by the broad categories of self-management intervention and support activity shown in Table 6 . Each coreviewer was allocated one or more category of studies to analyse. The lead reviewer then read each category of studies in the following order: face-to-face group support, online support, online information, information, psychological, lifestyle and ‘various’; within this, she read the studies in alphabetical order of first author rather than nominating ‘key’ papers. All included papers were analysed, rather than reading until saturation of concepts.

The lead reviewer and coreviewer independently completed matrices to report the second-order constructs and emerging third-order constructs for each paper (which for the lead reviewer were based on a more comprehensive line-by-line coding using NVivo). This facilitated the juxtaposing of metaphors and/or constructs alongside each other, leading to initial assumptions about relationships between studies.

Step 5: translating studies into one another

A defining element of metaethnography is the ‘translation’ of studies into each other, whereby metaphors, together with their inter-relationships, are compared across studies. Facilitated by discussions using the matrices of second- and third-order constructs, we translated studies firstly within types of support activity and then, secondly, across types.

The lead reviewer initially developed the constructs in relation to face-to-face support (the largest category of studies) and read other categories of studies with reference to this, using a constant comparison approach to identify and refine concepts. The ‘models’ function in NVivo was used to depict relationships between third-order constructs; this helped to develop the line-of-argument synthesis, which is discussed next.

Step 6: synthesising translations

Studies can be synthesised in three ways: 94

• reciprocal translation, where the findings are directly comparable
• refutational translation, where the findings are in opposition
• a line-of-argument synthesis, where both similarities and contradictions are found and translations are encompassed in one overarching interpretation that aims to discover a whole among the set of parts, uncovering aspects that may be hidden in individual studies.

Because we found similarities and contradictions, we developed a line-of-argument synthesis (rather than reciprocal or refutational translation) that encompassed four key concepts, each of which was based around a set of third-order constructs.

Step 7: expressing the synthesis

The output of the synthesis, that is communicating our third-order concepts and overarching line-of-argument synthesis, is described by Noblit and Hare 94 as ‘expressing the synthesis’ (p. 29). They state that ‘the worth of any synthesis is in its comprehensibility to some audience’ (p. 82), 94 emphasising the importance of communicating the synthesis effectively, being mindful of the intended audience and using concepts and language that are meaningful (and understandable). We worked to make the synthesis comprehensible by discussion with coreviewers and, critically, through involvement of the patient and public involvement (PPI) group. The synthesis is presented in Chapter 5 and will also be expressed through other dissemination activities, for example the SELF-MAN symposium ( www.self-man.com ), mini-manuals and journal publications.

We undertook several steps to enhance the rigour of our analysis. Authors’ themes and interpretations (second-order constructs) were independently extracted by two reviewers, each of whom additionally suggested their own interpretations of the study findings (third-order constructs).

We were influenced by a recent Health Technology Assessment metaethnography which found multiple reviewers offered ‘broad similarities in interpretation, but differences of detail’ (p. x). 131 We therefore treated the lead reviewer’s analyses as the ‘master copy’ and compared these with the coreviewers’ extractions and interpretations. Peer debriefing meetings were held between the lead reviewer and each coreviewer to discuss matrices of second-order and third-order constructs which facilitated the consideration of alternative interpretations.

The third-order constructs and line-of-argument synthesis were further refined at a full-day meeting (January 2014) attended by the lead qualitative reviewer and wider team of five coreviewers involved in coding, extraction, analysis and interpretation (PG, KH, LK, KM, CB).

We identified the need to be reflexive about our interpretations and recognised potential sources of influence on our interpretations; for example, two reviewers (PG, KH) identified having a ‘constructions of masculinity’ lens, and we agreed to focus the line-of-argument synthesis on interpretations offered by authors of studies being synthesised, rather than framing our interpretations around constructions of masculinity. We considered it a strength that the six reviewers involved reflected a wide range of backgrounds and perspectives. Although PPI colleagues were not involved in the coding process, the line-of-argument synthesis and four key concepts were discussed with the PPI group to ensure credibility.

• Public and patient involvement

The SELF-MAN research team worked with a specially constituted public and patient advisory group comprising men living with one or more LTCs who were involved in either running or attending a LTC support group in the north of England. Members were recruited via the research team’s existing networks. Stakeholders’ support groups were all condition-specific – arthritis ( n  = 1), diabetes ( n  = 1), heart failure ( n  = 2) and Parkinson’s disease ( n  = 1) – although some men lived with multiple LTCs. All stakeholders attended a welcome meeting prior to the commencement of the study to prepare them for the involvement in the research, and were provided with ongoing support and guidance by the chief investigator throughout the research process. Members were reimbursed for travel, expenses and time throughout the duration of the project (in line with current INVOLVE recommendations 138 ).

The overarching aims of PPI in the project were, first, to help ensure that the review findings spoke to the self-management needs and priorities of men with LTCs, and, second, to ensure the development of appropriate outputs that would have benefit and relevance for service users. A recognised limitation of our group was that stakeholder representation was drawn from face-to-face group-based support interventions.

The stakeholder group met on three half-days over the course of the 12-month project. On each occasion, the group provided positive affirmation that the project was being conducted in accordance with its stated objectives. In the first two meetings, the group offered feedback and advice to the investigative team on preliminary and emerging analysis of the qualitative data throughout the research process: specifically, the development of third-order constructs and the line-of-argument synthesis. Responding to their input, we made revisions to some of our interpretations, particularly in relation to the importance of physical aspects of environments in which interventions took place. The group’s input also highlighted the need for future research to address depression as a common and often overlooked comorbidity in men (see Chapter 7 , Recommendations for future research ), and that they welcomed recommendations for sustainability of support groups and improving communication within groups. When considering the key outcomes to be assessed in the quantitative review, stakeholders also recommended that emphasis should be placed on quality-of-life outcome measures when considering whether or not a self-management support intervention is effective.

In the final meeting, the stakeholder group provided detailed recommendations for the content of the Self-Manual: Man’s Guide to Better Self-Management of Long Term Conditions (not yet available). It advised that the guide should be rephrased from ‘how to’ self-manage to ‘how to better ’ self-manage because men may view themselves as already self-managing and therefore not identify with the former.

Six or seven stakeholders attended each meeting. The female partner of one of the men attended and contributed to discussions at each meeting. Members of the group each received reimbursement of travel expenses and a £150 honorarium for each meeting they attended. In the final meeting, the stakeholders provided feedback on their involvement in the research process overall, focusing on what was done well and what could be improved. Feedback indicated that most stakeholders had a positive experience, particularly valuing the opportunity to have their ‘voices heard’ and make a potential impact on future service delivery. Recommendations for improvements mostly centred on ensuring prompt reimbursement of expenses incurred in attending the meetings.

Included under terms of UK Non-commercial Government License .

• Cite this Page Galdas P, Darwin Z, Fell J, et al. A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN). Southampton (UK): NIHR Journals Library; 2015 Aug. (Health Services and Delivery Research, No. 3.34.) Chapter 3, Qualitative review methods.
• PDF version of this title (3.9M)

In this Page

Other titles in this collection.

• Health Services and Delivery Research

Recent Activity

• Qualitative review methods - A systematic review and metaethnography to identify... Qualitative review methods - A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN)

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

• Open access
• Published: 31 July 2024

When great responsibility comes with limited options: experiences and needs of older community-dwelling adults regarding accessing, understanding, appraising and using health-related information

• Sonja Stelly Gustafsdottir   ORCID: orcid.org/0000-0002-1790-4554 1 , 2 ,
• Lena Mårtensson   ORCID: orcid.org/0000-0002-6496-3661 3 ,
• Arun K. Sigurdardottir   ORCID: orcid.org/0000-0002-4287-5409 4 , 5 &
• Solveig A. Arnadottir   ORCID: orcid.org/0000-0002-3017-113X 2  

BMC Geriatrics volume  24 , Article number:  640 ( 2024 ) Cite this article

185 Accesses

Metrics details

In previous research, older adults have been associated with reduced levels of health literacy (HL) influenced by a range of contextual factors. To optimise HL, it is essential to better understand the interactions between the ageing process and both personal and environmental factors as perceived by older adults. This study aimed to explore the experiences and needs of older community-dwelling adults when accessing, understanding, appraising and using health-related information.

An explorative, qualitative design was used within the social constructivism framework. Semi-structured individual interviews were conducted with 20 adults aged 70–96 living at home in three areas in Northern Iceland. The transcribed interviews were constructed into categories and subcategories using qualitative content analysis.

Four categories emerged. “Expectations for responsibility” describes the experience that individuals are responsible for taking care of their health, including accessing, understanding, appraising and using information and services, showing initiative and keeping needed communications active. “A gap between expectancy and ability/context” includes experiences while taking responsibility for expectations not aligning with skills/situations, creating information gaps. “Finding one’s own ways” comprises various adapted ways to access, understand, and use information and services. “Bridging the gap” describes experiences of needing shared responsibility and more manageable options to enable reasoned health-related decisions and navigation in the healthcare system.

Conclusions

The participants valued and took full responsibility for accessing, understanding, appraising and using information and services as part of a social norm; however, they experience information gaps. They request shared responsibility by being provided with fundamental health-related information as a vital step in making reasoned health-related decisions and navigating the healthcare system. They also request more inclusive and accessible service opportunities to bridge the gaps and facilitate HL. It is necessary to critically address, at a systematic level, the conflict between expected individual responsibility and the existence of options to act upon this responsibility. In matters of health, health services and HL, the need to analyse and confront structural disadvantages experienced by older adults is highlighted.

Peer Review reports

Introduction

Health literacy (HL) refers to both the personal skills and social resources needed for individuals and communities to access, understand, appraise and use information and services to make reasoned health-related decisions and to navigate in the healthcare system [ 1 , 2 ]. HL is therefore considered essential to maintaining and improving quality of life throughout the life course [ 3 ]. With advancing age, older adults may require more frequent interactions with health-related information. Consequently, the importance of addressing HL concerning older adults has been emphasised [ 4 ], particularly because of population projections that indicate a global increase in the number of older adults in the future and the need to enhance health promotion for this group [ 5 ].

In Europe, an effort has been made to measure HL levels, for example, with the European Health Literacy Survey Questionnaire (HLS-EU-Q), among the general population. Limited HL is associated with social and socioeconomic conditions, particularly lower levels of education, income, low social status and older age [ 4 , 6 , 7 , 8 ].

Acknowledging HL as an interaction of individual skills within a social context, it is essential to look beyond the personal level and include the social structures in which people live. That is, to better understand the influence of the situations in which people are required to use their HL skills and capabilities [ 9 , 10 ]. There is, for example, a heightened focus on the organisational context of HL, the health system’s demands and the complicated information environment in a modern world [ 11 , 12 ]. However, at the same time, it is essential to recognise the complexity of the social context of HL [ 9 , 10 ]. This has, for example, been addressed in research focusing on experiences related to health information among socioeconomically disadvantaged adults in Switzerland [ 13 ], among refugees in Sweden [ 2 ] and as part of information literacy in everyday life among people aged 47–64 [ 14 ] and 57–70 in Australia [ 15 ].

Research findings concerning older adults in Iceland echo this complex interaction between the ageing process, HL and both personal and environmental factors. Notably, HL has been connected to the personal factors of age in years, education level, income, resilience and depression and the environmental factors of means of transport and perceived access to healthcare and medical service [ 16 ]. These factors seem to play an important role in HL; however, further information is needed to comprehensively understand this dynamic interaction between older adults, HL and their context. In contrast to using quantitative measurements in relation to HL as is prominent, a qualitative perspective is needed to gain a deeper understanding of the matter. Therefore, this study aimed to explore the experiences and needs of older community-dwelling adults concerning accessing, understanding, appraising and using health-related information.

Study design

This qualitative study within the social constructivism framework sought to understand the specific contexts in which people live [ 17 , 18 ]. An explorative design was used to find and create knowledge of the focused and little-studied phenomena [ 19 ]. We conducted individual semi-structured interviews to generate qualitative data, get insights into the lives of older adults and establish knowledge [ 20 ].

The research group consisted of three Icelandic researchers (SSG, SAA and AKS), a Swedish (LM) researcher and an Icelandic senior citizen (AS). The four researchers created an interdisciplinary team of occupational therapists (SSG and LM), a physiotherapist (SAA) and a nurse (AKS) as professors/researchers (LM, SAA and AKS) and a PhD student (SSG). The group’s expertise, research and lived experience lie within ageing, daily living, gerontology, HL, health promotion, qualitative research and urban/rural settings.

This study is reported following the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist [ 21 ]. An application for ethical approval was sent to the Icelandic National Bioethics Committee. The committee deemed permission not necessary according to Icelandic law on scientific research in health (VSN-21–009 based on VSNb2016060007/03.01).

Participants and setting

Participants in this study were purposefully selected from 175 partakers in a previous quantitative cross-sectional study on HL. That study was based on a stratified random sample from the national register of community-dwelling people 65 years and older in one urban town and two rural areas in Northern Iceland [ 16 , 22 ].

To get as broad a perspective as possible, the selection criteria for this study were based on the aim of interviewing older people with various backgrounds regarding place of living, age, gender, education, means of transport and distance from services. The selection procedure was conducted in three steps, as shown in Table  1 . In the first step, potential participants were sorted by partakers’ numbers from the previous quantitative research. They were placed into a matrix list based on five to six determining factors, with a sixth factor being considered for those living in rural areas. Considering the amount of needed information, that some people might not be reached and some might decline participation, the matrix list included 69 from 175 previous partakers, with many categorised with the same factors. In the second step, previous partakers’ numbers and the names of potential participants were connected. Information recorded at the Register Iceland database on a) social security number, b) place of living and c) a registered telephone number accessible through an open website were matched. This information could not be paired for 21 persons, leaving 48 on the potential participants’ list. In the third and last step, 20 people on the list were contacted for participation. They all agreed, consented, and were subsequently interviewed.

All participants, 11 women and nine men were born and raised in Iceland except for one individual who, despite not being native, had resided in the country for decades. Their birth years ranged from 1926 to 1952, and the median age was 76,6 years. Seven had elementary education, eight a secondary or trade school education and five a university degree. Agriculture, education, trade, healthcare and homemakers were the main occupation fields.

In three interviews, the spouse was present. In one case, the participant had early-stage Alzheimer’s, so in cooperation with the couple, it was decided that the spouse would play the roles of support, memory and voice. In the other two cases, both in rural settings, the spouse of the participant was present in the kitchen, where the interview was conducted as a part of the culture in place. The spouses were not direct participants in the interview; however, they added information when, for example, asked to recall a process, names or times.

Potential participants were sent an invitational letter by mail and subsequently contacted (by SSG) by telephone and invited to participate. Data were collected over one year, from January 2022 to January 2023. SSG conducted all of the interviews in Icelandic at the participants’ chosen place. The interviews were audio-recorded and lasted from 30 to 65 min, with an average of 46 min. The recordings were deleted after transcription. An interview frame designed for this study was used, which included a) opening questions about the length of time living in their current area, main occupation and preferred pseudonyms for confidentiality; b) questions about HL, which asked the participants to describe their experience of accessibility, clarity and usefulness of health-related information and services; and c) an opportunity to add any information.

Data analysis

The interviews were transcribed verbatim and analysed using content analysis as described by Graneheim and Lundman [ 23 ] and Graneheim, Lindgren and Lundman [ 24 ]. The method offers researchers different epistemological positionings with various levels of abstraction and degrees of interpretation, depending on the study aim and data quality. It is, for example, applicable when knowledge is believed to be socially constructed [ 24 ]. As reflexivity was considered an essential part of the whole process, the analysis was conducted with a team of all authors. Although SSG and LM mostly did the main work because of their pre-understanding of the research area and the method used for analysis, all the authors met in working meetings at each step of the analysis process, as described below. These meetings were used for reflection on the empirical data, the potential influence of preconceptions and the emerging findings from SSG and LM. In addition, a reflection from AS with a lived experience of the matter was sought in each step. Throughout the analysis process, work was carried out in Icelandic and English. Transcriptions were entered into the data management software NVivo 11 [ 25 ] for data storage, organisation, and coding. However, the team encountered difficulties in sharing information using the software, which resulted in the analysis being conducted manually in a Word document.

In the first step, all the authors read the interviews several times to understand the content. The three Icelandic researchers read the material in their native tongue, and the Swedish researcher used an English Google-translated version. The interviews were discussed both as a whole and in specific parts, where meaning units and potential content areas in consideration of the study’s objective were identified.

In the second step, meaning units were identified according to the aim of the study and condensed into descriptions close to the text, preserving the core meaning, abstracted and labelled with codes. Further abstraction occurred as subcategories and categories emerged from the condensed content based on patterns or commonalities. Similarities, differences and connections between and within the content were reflected upon and sorted. Constant comparison was used to clarify meanings, comparing data with codes and codes with codes.

In the third and last step, further analysis took place when the descriptive content of the preliminary categories was formulated by going back and forth and checking consistency between the categories, their content and the empirical data. The emerged core meanings were validated by contextualizing the meaning units in the individual interviews and the data as a whole. The 20 interviews provided insightful data to answer the purpose of the study, which was to explore the experiences and needs of older community-dwelling adults concerning accessing, understanding, appraising and using health-related information.

Based on manifest content, four qualitative categories emerged from the experiences and needs of older community-dwelling adults. Each category is independent, yet interconnected with the others, as shown in Fig.  1 . “Expectations for responsibility” describe the experience that the individual should be responsible for taking care of their own health, including accessing, understanding, appraising and using information and services, as well as showing initiative and keeping needed communications active. “A gap between expectancy and ability/context” includes experiences while taking responsibility for expectations not aligning with one’s own skills/situations. “Finding one’s own ways” comprises various adapted ways to access, understand, appraise and use information and services due to a misalignment between expectations for responsibility and the individual's ability or context. “Bridging the gap” describes experiences of needing shared responsibility and more manageable options to optimise reasoned health-related decisions and navigation through the healthcare system. Significant quotations are provided to illustrate the empirical foundations of the subcategories. Table 2 provides information about the manifest content from the analysis.

Interplay of categories and subcategories for older adults' experiences and needs related to health literacy. The figure shows the interaction and tension between the categories “Expectations for responsibility” and “A gap between expectancy and ability/context” that together create limited options and push for individual action in “Finding one's own ways” when accessing, understanding, appraising, and using health-related information. Jointly, these three categories, including their subcategories, call for needed actions in “Bridging the gap”, presented in the green box

Expectations for responsibility

This category describes participants’ experiences regarding the predominant expectation that each individual should carry the responsibility of taking care of their own health, including accessing, understanding, appraising and using health-related information as well as showing initiative and keeping needed communications active. The category is divided into two subcategories based on how this expectation is described: directly from the person and indirectly from the information providers. The subcategories were named “ Personal expectations ” and “ Environmental expectations ”.

Personal expectations

This subcategory describes the experience that the person was expected to be responsible for their health and health-related matters—that is, to be their own health manager. The person should know best what they need and therefore be responsible for achieving, understanding, appraising and using information. This expectation was accepted as part of being independent and acknowledging the increase in general knowledge, making people more educated about health matters. By not taking responsibility as one’s health manager, opportunities for health and welfare information might be lost, and then the person would be the only one to blame.

“You get the information you need, you just look for it… so you have nothing to complain about but yourself” (if you have missed information) (Thorunn, 76-year- old woman).

Environmental expectations

The responsibility for taking care of one’s own health was also experienced as an unspoken expectation from health-related information and service providers, who often only deliver information if requested. In these circumstances, the individual needs to take the initiative to look for the information and services that are needed and relevant on each such occasion. If opportunities to manage one’s own health were lost, this was because of a lack of responsibility of the persons rather than the information provider.

“I did not know… usually it is the case that you have to look for information” (Kara, 70-year-old woman).

A gap between expectancy and ability/context

This category describes participants’ experiences of being unable to live up to the expectations of being responsible for accessing, understanding, appraising and using health-related information to manage their health. The category is divided into three subcategories based on descriptions of different kinds of gaps between the expectancy and one’s own ability/context, although often interlinked, which are named “ Digitalisation gap ”, “ Personal contact gap ”, and “ Navigation gap ”.

Digitalisation gap

This subcategory describes the experiences of being unable to access and use information as expected and navigating within and between the health and welfare systems because of the increased use of computers and the internet, that is, digitalisation. Although digital development was generally viewed positively, it was expressed that all the changes were happening so fast, leaving many unable to keep up. For those needing more than general information or not having all the proper equipment or the ability, digital technologies were creating a significant gap in information and services.

“You know, I am back from ancient times. I have no computer and no phone to Google and nothing, so I am completely… so many things that you cannot do unless you have a computer… all the information” (Dora, 96-year-old woman).

Personal contact gap

With the increased use of digital technology, there was also the experience of a decrease in personal and direct contact. This combination created an even wider gap between expectations of taking responsibility and one’s own ability/context. This gap consisted of being unable to use entirely the formal digital ways to access, understand, appraise and use information and, simultaneously, the conventional and valued forms of person-to-person contact being limited. Being without a key person within the health and welfare systems to contact was described as being lost and not knowing what information to look for, where and what options were current or applied to them. This contact with a key healthcare person was significant in the case of illnesses. Although generally satisfied with hands-on service, with no one knowing the health history and situation of the older person or the possibilities in the service system, there was no way to safely navigate or coordinate the necessary information and actions when needed.

“… I need to get someone I trust. I do not want to end up with a new person in every conversation and say the same thing over and over and over again. After the fifth time, you think 100 times over whether to call again… Everyone wants to assist you, but can’t because they do not know you” (Hanna, 80-year-old woman).

Navigation gap

A gap in navigation while taking responsibility for accessing, understanding, appraising and using health-related information was experienced as a result of the general complexity within and between health-related services, particularly in knowing what information to find and where. This gap in navigation was fuelled by the digitalisation gap and the personal contact gap. The existence of long, complicated, and unclear communication channels, disconnection between organisations, and unclear service provision or division between entities, such as the local municipality and the state, often result in difficulties finding information or some information getting lost along the way. These were expressed as daunting, never sure of being on the right navigation course, always showing initiative and only relying on persistence not to give up. Perceived by participants as confusing, health-related service systems were described as not being made for the older service user and made to drive them away.

“…this is uncomfortable because you sometimes get the impression that the system does not care… and then you think you are somehow alone if something happens. Why isn’t it better? Maybe that is why senior citizens get the impression that you are a bit set aside. It is tiring always to have to push yourself somehow through” (Sigrun, 78-year-old woman).

Finding one’s own ways

This category describes the experiences of finding one’s own ways to live up to the expectancy of being responsible for accessing, understanding, appraising and using health-related information. It is divided into three subcategories based on descriptions of the different ways used to adapt: “ Rely on oneself ”, “ Rely on spouse, family and friends ”, and “ Using personal relationships ”.

Rely on oneself

This subcategory describes the experience of relying on oneself while managing health-related information. It was described as using knowledge through former work experience from the health and welfare service, watching parents age or even taking care of them and building on information and service from that time. Having some idea about what service is available and where to start looking for further information was expressed. This includes having enough knowledge to know where to look for information and how the services operate, given that little has changed.

“I know the operation (at the former workplace) well enough that I would look for the service, if I needed home care or something like that, I know how to do it” (Nina, 80-year-old woman).

Rely on spouse, family and friends

Finding one’s own ways based on the experience of relying on the spouse, family and friends regarding health-related information was described. In situations in which the participant could not use computer technology fully or at all, but the spouse could, he or she was valued as essential and even the reason for being able to live in place. Help from grown-up children or acquaintances was also mentioned, and they considered themselves lucky to have people around to help, stating that this was not the case for everyone.

“Our daughter … is extremely good at helping. I do not know what we would do if we lost touch with her” (Fannar, 72-year-old man).

Experiences of getting information about available services and where to turn in need at gatherings organized by local senior non-governmental organisations (NGOs) were also described. Also, when getting together, friends shared information on where to turn in need and hands-on experiences.

Use personal relationships

Finding one’s own ways by using personal relationships or acquaintances with health and welfare professionals was described by some as being, at times, necessary to access information or services by using this kind of relationship. This required using informal methods rather than formal ones when no other means seemed possible.

“He (the general practitioner) was always on vacation or busy or not reachable … so I called my son (who is a medical doctor) and said now you have to help me” (Dora, 96-year-old woman).

Others described it as a common way to use personal connections regarding health-related information, especially in rural areas where “everybody knows everyone”. Considerations of being very lucky to have this kind of relationship and being able to use this informal way were expressed.

Bridging the gap

This category describes the experience of what is needed to access, understand, appraise and use health-related information to be more able to take care of one’s own health. It is divided into two subcategories based on descriptions of different needs: “ Shared responsibility ” and “ Manageable options ”.

Shared responsibility

This subcategory describes the experience of needing shared responsibility by being provided with the necessary fundamental health-related information. Although accepting the expectations of being responsible for achieving information as a part of taking care of their own health (category “Expectations for responsibility”) the experience also revealed that to do so, fundamental knowledge of what information and services exist and are current is required. To find information about services, the person first needs to know what opportunities and resources are available.

“ I really expect this (information) to be handed to me when I reach the age, but not that I have to run after it” (Hanna, 80 year-old woman).

Some of the fundamental information on health-related matters was described as being provided by local senior NGOs and highly valued as such; however, at the same time, it was questioned who should be responsible for providing older adults with this information.

Manageable options

In addition to needing the provision of fundamental health-related information to bridge the gap, this subcategory describes the experience of what kind of information access is required and in what way accessed. Manageable options refer to an accessible overview of opportunities and resources that are available and current, both locally and nationwide. A clear venue for quality and reliable information is necessary, which was possible during the COVID-19 pandemic, so there is a precedent. Also, getting valuable and more relevant information is needed.

“Why do you always have to be in such a terrible shape to get information and service? … I think there needs to be a little more about everyday things. If you’re taking care of yourself, advice is needed on the best way to do this” (Nina, 80-year-old woman).

Access to information and services must align with diverse abilities/contexts. More options than mainstream digitalisation to access information and navigate through service systems need to be available. In rural areas, experiences of information being delivered more according to the ability and context of people were described, however, as being more the personal decisions of the staff rather than an embedded system ideology. Manageable options also include offering in-person support for those who require more introduction, instructions or assistance when accessing, understanding, appraising and using information.

The findings of this study among community-dwelling older adults revealed four separate, but interconnected, qualitative categories. The category “Expectations for responsibility” describes the experience that the person, the individual, should be responsible for taking care of their health, including accessing, understanding, appraising and using information and services. However, difficulties in doing so are revealed in the category “A gap between expectancy and ability/context” and include experiences, while taking the responsibility of expectations to do so are not in line with skills/situations. The consequences are information gaps that arise. The category “Finding one’s own ways” comprises various adapted ways to access, understand, appraise and use information and services. Although accepting the expectations that the individual should be responsible for taking care of their health, the category “Bridging the gap” describes experiences of needing responsibility to be shared and more manageable options to optimise reasoned health-related decisions and navigation in the healthcare system.

The category “Expectations for responsibility” is the base for the categories “A gap between expectancy and ability/context” and “Finding one´s own ways”. It describes the responsibility that participants experience in accessing, understanding, appraising and using health-related information and services as part of being their own health managers. This view is fuelled by and intertwined with personal expectations and messages from the environment that seem to be a part of social norms. This experience echoes, in a way, neoliberal ideology, with its economic and political focus on individualism and autonomy. It includes the idea that people should have the right and responsibility to make their own choices, which inevitably shapes healthcare delivery systems [ 26 , 27 ]. Reflecting this upon Iceland, although generally considered a part of the Nordic welfare states, the rise and promotion of neoliberalism in the country has shaped the economy and politics of health and welfare since the late 1970s [ 28 ].

Although the findings from this study indicated that the participants accept the expectation of being responsible for their health and value being their own health managers, they also revealed that this expectation was often not in line with their skills/situations. The result was the emergence of information gaps limiting their options to be responsible and make informed health decisions. The “A gap between expectancy and ability/context” category describes three interlinked subcategories: digitalisation , personal contact and navigation gaps . Numerous studies have reported challenges among older adults to participate in or benefit from the growing digitalisation, known as the “digital divide” or “grey digital divide” [ 29 , 30 ]. Research findings from Iceland also highlight this information gap. Palsdottir [ 31 ] has reported an increased frequency of online health information seeking among people 68 years and older from 2002 to 2012. However, the usefulness of that information, including websites by the health care system or health specialists, did not increase. A study on perceived barriers to health information among people 60 years and older also reveals hindrances in the availability of information and the ability to seek and find it [ 32 ]. This divide is considered to marginalise older adults, among other groups, who are most likely to become excluded from the benefits of digital technologies [ 30 ]One solution to the digital transformation that healthcare is undergoing [ 29 ], and in line with the expectation of individual responsibility, would be to modify HL by interventions aimed at strengthening the digital skills or competencies of individuals through education. However, Bittlingmayer and Sahrai [ 33 ] drew attention to what might happen if increased education is challenging to manage – for example, in the case of disability. Although older adults are a heterogeneous group, this perspective could be reflected, for instance, in the normal ageing process. Lifelong learning [ 5 ] should always be an option; however, how health services meet the complex needs of people as their own health managers needs to be addressed. Another angle regarding these experienced information gaps is that limited HL has generally been related to lower levels of education [ 4 , 6 , 16 ]. In this study, however, the community-dwelling participants’ education level is relatively high, with most having a secondary or a university degree. Perhaps this echoes the dynamic and complex interaction between various personal and environmental contextual factors acting and interacting as barriers or facilitators of HL.

Because of the gaps experienced in health-related information, the participants need to adapt and find other ways to manage. They do this by, for example, relying on people close to them like spouses, children, grandchildren and friends, as described in the category “Finding one’s own ways”. Concerning this adaptation, the resilience and resourcefulness of the participants seem to play an important role; it also identifies the importance of social connections or networks. Making reasoned health-related decisions and navigating the healthcare system can, therefore, build on if you have someone in your life willing and able to help. Findings from this research indicated that, in some cases, this support is provided by healthcare staff, even when not on the job. ers. The importance of relatives as an adapted or alternative strategy while dealing with health information was one of four main themes in a study among socioeconomically disadvantaged adults in Switzerland [ 13 ]. This supports the importance of social connection and support for those experiencing health-related information gaps. Although matters of caregiving and the share of informal/unpaid carers, often female family members or friends [ 34 ], are beyond the scope of this research and will not be addressed further, the effects of individualism and healthcare delivery systems’ expectations of responsibility cannot be underestimated. The local senior NGOs in Iceland also seem to play an important role in providing relevant health-related information, for example about rights and available services. However, to participate in gatherings these organisations provide or to receive most of the information, people must become members and pay an annual fee [ 35 ].

In the category “Bridging the gap”, the participants ask for two things to make the passing of the experienced information gaps easier, presented in the subcategories “ Shared responsibility ” and “ Manageable options ”. Although the findings from this study indicated that the participants accept the expectation of responsibility for their own health, they also revealed a contradiction. Namely, without knowing what information and services exist and are current, this expectation can sometimes be hard to live up to, or even be impossible. Shared responsibility in providing older adults with more fundamental health-related information seems a vital preliminary step for them to access, understand, appraise and use information. Access to healthcare is generally considered a multifaceted concept consisting of the interaction between the accessibility of services and the abilities of people [ 36 ]. One of the five identified dimensions of accessibility is the approachability of services. This dimension includes making services known and reachable to individuals, along with the necessary individual skills to identify the need for these services [ 36 ]. Yet again, the complexity of HL is brought to light, and the question is raised where this line between individual versus service responsibility is drawn and, more importantly, who decides. Organisational health literacy (OHL) is an evolving concept [ 11 ], especially in the wake of COVID-19, which has transformed the healthcare service [ 37 ]. It is described as an effort to transform health-related services to make it easier for people to navigate, understand and use information and services to look after their own health and address the implementation of policies, practices and systems. This concept underpins the idea that HL does not merely depend on the abilities of individuals [ 11 , 38 ]. Neoliberal policies, with their emphasis on economic value, have often been criticised for negatively impacting access to healthcare by not addressing the structural disadvantages experienced by certain population groups [ 39 ]. Furthermore, these policies are considered to contribute to the negative viewing of individuals who are not in the workforce, perceiving them as potentially financially burdensome [ 39 ].

Also, older adults may experience decreased functioning over time due to the natural ageing process. This decline can affect their ability and capacity to access, understand, appraise and use health-related information. Focusing on individual responsibility can lead to compromised access to and use of information and services. Therefore, the effect of these policies on fuelling ageism in viewing older adults as a burden must be considered. Ageism has been estimated to cost societies vast amounts [ 40 ], and in the United Nations action plan Decade of Healthy Ageing 2021–2030 [ 5 ], one of the identified areas for action is connected to changing negative views and actions towards age and ageing.

The findings from this study indicated that the participants not only require shared responsibility in the form of being provided with fundamental health-related information in the category of “Bridging the gap”. The subcategory “ Manageable options ” reflects the request for information that is approachable, acceptable, appropriate, and available. For example, this study indicates that older adults have a generally positive view of digital development as a part of the future. This finding is also reflected in a study on technology use for health information based on a randomized sample of older Icelanders [ 41 ]. However, it seems to be an issue of design, delivery, instructions and support, bringing us to service user participation and inclusion. One of the identified areas for action in the United Nations action plan, Decade of Healthy Ageing 2021–2030 [ 5 ], aims at enabling older people to continue to do the things that they value and support the inclusion of their voices not only as service beneficiaries but also as agents of change. The focus is on the abilities of older people and person-centred integrated care and primary health services. Brach et al. [ 42 ] introduced the 10 attributes of a health-literate healthcare organisation (HLHCO). The attributes are based on the OHL concept [ 11 ] to deliver person-centred healthcare and tackle system-level factors enabling people to access, understand, appraise and use health-related information. One of the 10 attributes emphasises the importance of including the voices of consumers in the design process, implementation and evaluation of health information and services [ 42 ]. This specific attribute, engagement and support of service users, has been recognized as one of the most prevalent topics of OHL [ 38 ].

Furthermore, in a framework for strengthening the health system’s capacity regarding HL, one of the eight suggested action areas focuses on people-centred services based on user engagement and enabling environments [ 43 ]. In this study, the participants indicated that they value being their own health managers and take full responsibility for accessing, understanding, appraising and using health-related information, as expected, as part of social norms. However, the lack of options to fulfil this expectation implies that healthcare delivery systems do not always meet the needs of older adults to act on it.

Strengths and limitations

This qualitative exploratory study aimed to gather information about the experiences and needs of older community-dwelling adults concerning accessing, understanding, appraising and using health-related information. One of the strengths of this study is that it gives older adults living at home a platform to be heard. By selecting potential participants purposefully with different backgrounds regarding the place of living, age, gender, education, means of transport and distance from services, variations in experiences were sought. The generalisability of the results was affected by participants being restricted to living in Northern Iceland and including individuals with similar cultural backgrounds. It should, however, be kept in mind that close similarities may exist between Iceland and other northern geographical areas of the world where the culture is labelled Western.

The possible effects of having a spouse present during three of the 20 interviews must be mentioned. Their presence was considered culturally relevant in rural areas in the sense of greeting visitors at home. The spouses also acted as a support and facilitated communication, such as for one participant with early-stage Alzheimer’s disease. Memory loss is most often a reason for exclusion from research. However, gender roles and the power balance between couples must be considered, which might have affected the conversations. One interview took place via Zoom. While this may not align with our main findings, older adults’ technological skills vary. In times of often hard confinement and isolation of older people during the COVID-19 pandemic, by preparing the interview setting well, the wishes of this participant to meet on Zoom could be met.

Clear categories emerged based on evident patterns, consisting of direct content, minimal interpretation and remaining close to the original text. In content analysis, the researcher must know the context. Having four interdisciplinary researchers with stated expertise partaking in the data analysis process contributed to the credibility of this research. Although two researchers conducted the primary analysis, regular meetings with all authors at every step of the process were used for reflection on possible preconceptions and consistency between empirical data and the emerging categories and their content. Including a senior citizen with lived experience on the research team further enhanced the credibility of this research. However, the involvement of an older adult in the earlier stages of the research is an aspect for consideration in future studies.

Working on data in Icelandic and English can be both a strength and a limitation. A strength regarding reasonability and accuracy as a thorough evaluation of the meaning and use of words during the translation process took place. A limitation in the sense of possibly misrepresenting the participants’ expression in the translation process from Icelandic to English, although three of four researchers are fluent in both languages, should minimise that risk.

The participants in this study experienced expectations of being responsible for accessing, understanding, appraising and using health-related information as part of acting as their own health managers. Although valuing and accepting these expectations, limitations regarding living up to them were revealed because such expectations were often not in line with their skills/situations, despite having a relatively high education level. Information gaps, therefore, arise due to digitalisation, limited personal contact and general navigation complexity within and between health-related services. Therefore, approachable fundamental health-related information, current and quality checked, and inclusive service opportunities are needed to bridge the resulting gaps. It is necessary to critically address the possible influences of politics regarding the views on individual responsibility at a systematic level in matters of health and HL. Such action should analyse if and how those principles shape attitudes, social norms and health services and confront structural disadvantages experienced by population groups. Access to information and services must be viewed beyond availability and include the approachability, acceptability and appropriateness of service users with various abilities and contexts. The findings from this study reflect participants’ experiences of bearing most of the responsibility as their own health managers, while simultaneously having limited choices in acting on it. Policymakers are therefore encouraged to develop services that enable older adults to make reasoned decisions about health and navigate healthcare services in an effective way.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author upon reasonable request.

Abbreviations

Consolidated Criteria for Reporting Qualitative Research

Coronavirus disease 2019

• Health literacy

Health literate health care organisation

European Health Literacy Survey Questionnaire

Non-governmental organisations

Organisational health literacy

Mårtensson L, Hensing G. Health literacy - a heterogeneous phenomenon: a literature review. Scand J Caring Sci. 2012;26(1):151–60.

Article   PubMed   Google Scholar  

Mårtensson L, Lytsy P, Westerling R, Wångdahl J. Experiences and needs concerning health related information for newly arrived refugees in Sweden. BMC Public Health. 2020;20(1):1–1044.

Article   Google Scholar  

Sørensen K, Van den Broucke S, Fullam J, Doyle G, Pelikan J, Slonska Z, et al. Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health. 2012;12(1):80.

Article   PubMed   PubMed Central   Google Scholar  

Dietscher C, Pelikan J, Bobek J, Nowak P, World Health Organization. The Action Network on Measuring Population and Organizational Health Literacy (M-POHL): a network under the umbrella of the WHO European Health Information Initiative (EHII). Public Health Panor. 2019;1(5):65–71.

Google Scholar  

World Health Organization. United Nations decade of healthy ageing, 2021–2023. 2020. Available from: https://www.who.int/publications/m/item/decade-of-healthy-ageing-plan-of-action .

Sørensen K, Pelikan JM, Röthlin F, Ganahl K, Slonska Z, Doyle G, et al. Health literacy in Europe: comparative results of the European Health Literacy Survey (HLS-EU). Eur J Public Health. 2015;25(6):1053–8.

Nutbeam D, Lloyd JE. Understanding and responding to health literacy as a social determinant of health. Annu Rev Public Health. 2021;42(1):159–73.

Stormacq C, Van den Broucke S, Wosinski J. Does health literacy mediate the relationship between socioeconomic status and health disparities? Integrative review Health Promo Int. 2019;34(5):e1–17.

Pitt R, Davis T, Manganello J, Massey P, Okan O, McFarlane E, et al. Health literacy in a social context: a meta-narrative review. In: Okan O, Bauer U, Levin-Zamir D, Pinheiro P, Sørensen K, editors., et al., International handbook of health literacy. Bristol: Policy Press; 2019. p. 665–88.

Chapter   Google Scholar  

Sentell T, Pitt R, Buchthal OV. Health literacy in a social context: review of quantitative evidence. Health Lit Res Pract. 2017;1(2):41.

Farmanova E, Bonneville L, Bouchard L. Organizational Health literacy: review of theories, frameworks, guides, and implementation issues. Inquiry. 2018;55(1):46958018757848.

PubMed   Google Scholar  

Frederiksen N, Wångdahl J. Health literacy in the Nordic countries : Not only a determinant of health, but also a tool for health promotion. Nordic Welfare Centre. 2022. Available from: https://nordicwelfare.org/en/publikationer/health-literacy-in-the-nordic-countries-not-only-a-determinant-of-health-but-also-a-tool-for-health-promotion/ .

Stormacq C, Bachmann AO, Broucke SVd, Bodenmann P. How socioeconomically disadvantaged people access, understand, appraise, and apply health information: A qualitative study exploring health literacy skills. PLOS ONE. 2023;18(8):e0288381.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Yates C, Partridge H, Bruce C. Learning wellness: how ageing Australians experience health information literacy: The Australian Library J. 2009;58(3):269–85.

Yates C. Exploring variation in the ways of experiencing health information literacy: A phenomenographic study. Libr Inf Sci Res. 2015;37(3):220–7.

Gustafsdottir SS, Sigurdardottir AK, Mårtensson L, Arnadottir SA. Making Europe health literate: including older adults in sparsely populated Arctic areas. BMC Public Health. 2022;22(1):511.

Crotty MJ. The foundations of social research. 1st ed. London, England: SAGE Publications; 1998.

Creswell JW, Poth CN. Qualitative inquiry & research design: choosing among five approaches. 4th ed. Thousand Oaks: Sage; 2018.

Stebbins RA. Exploratory research in the social sciences. Thousand Oaks: SAGE Publications Inc; 2001.

Book   Google Scholar  

Kvale S, Brinkmann S. InterViews: learning the craft of qualitative research interviewing. 3rd ed. Thousand Oaks: SAGE Publications Inc; 2008.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

Sigurdardottir AK, Kristófersson GK, Gústafsdóttir SS, Sigurdsson SB, Arnadottir SA, Steingrimsson JA, et al. Self-rated health and socio-economic status among older adults in Northern Iceland. Int J Circumpolar Health. 2019;78(1):1697476.

Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today. 2004;24(2):105–12.

Article   CAS   PubMed   Google Scholar  

Graneheim UH, Lindgren B, Lundman B. Methodological challenges in qualitative content analysis: a discussion paper. Nurse Educ Today. 2017;56:29–34.

Lumivero. NVivo (Version 11). 2015. www.lumivero.com .

Barnett P, Bagshaw P. Neoliberalism: what it is, how it affects health and what to do about it. N Z Med J. 2020;133(1512):76–84.

Nordgren L. Mostly empty words - what the discourse of “choice” in health care does. J Health Organ Manag. 2010;24(2):109–26.

Ólafsson S. Political turmoil in Iceland. Nordiques. 2016;1(32):29–42.

Mubarak F, Suomi R. Elderly forgotten? Digital exclusion in the information age and the rising grey digital divide. Inquiry (Chicago). 2022;59:469580221096272.

Pirhonen J, Lolich L, Tuominen K, Jolanki O, Timonen V. “These devices have not been made for older people’s needs” – Older adults’ perceptions of digital technologies in Finland and Ireland. Technol Soc. 2020;62:101287.

Pálsdóttir Á. Senior citizens, media and information literacy and health information. In: Kurbanoglu S, Boustany J, Špiranec S, Grassian E, Mizrachi D, Roy L, editors. Information\ Literacy: Moving Toward Sustainability. Switzerland: Springer International Publishing AG; 2016. p. 233–40. ECIL 2015, CCIS 552.  https://doi.org/10.1007/978-3-319-28197-1_24 . Available from https://link.springer.com/book/10.1007/978-3-319-28197-1 .

Pálsdóttir Á. Older Icelanders’ experience of barriers to health information: association with age, sex, and education. Human Aspects of IT for the Aged Population. Design, Interaction and Technology Acceptance Cham: Springer International Publishing. p. 567–583.

Bittlingmayer UH, Sahrai D. Health literacy for all? Inclusion as a serious challenge for health literacy: the case of disability. In: Okan O, Bauer U, Levin-Zamir D, Pinheiro P, Sørensen K, editors. International handbook of health literacy. Bristol: Policy Press; 2019. p. 689–703.

Organisation for Economic cooperation and Development. Health at a glance 2019: OECD indicators. 2019. Available from: https://doi.org/10.1787/4dd50c09-en .

Landsamband eldri borgara (Association for older adults). Available at: https://www.leb.is/um-leb/ .

Levesque J, Harris MF, Russell G. Patient-centred access to health care: conceptualising access at the interface of health systems and populations. Int J Equity Health. 2013;12(1):18.

Sentell T, Foss-Durant A, Patil U, Taira D, Paasche-Orlow MK, Trinacty CM. Organizational health literacy: opportunities for patient-centered care in the wake of COVID-19. Qual Manag Health Care. 2021;30(1):49–60.

Bremer D, Klockmann I, Jaß L, Härter M, von dem Knesebeck O, Lüdecke D. Which criteria characterize a health literate health care organization? – a scoping review on organizational health literacy. BMC Health Serv Res. 2021;21(1):1–664.

Sakellariou D, Rotarou ES. The effects of neoliberal policies on access to healthcare for people with disabilities. Int J Equity Health. 2017;16(1):199.

World Health Organization. Global report on ageism. 2021. Available from: https://www.who.int/teams/social-determinants-of-health/demographic-change-and-healthy-ageing/combatting-ageism/global-report-on-ageism .

Pálsdóttir Á. The use of new information and communication technology for health information among older Icelanders. Human Aspects of IT for the Aged Population. Healthy and Active Aging Cham: Springer Int Publishing. p. 354–364.

Brach C, Keller D, Hernandez LM, Baur C, Parker R, Dreyer B, et al. Ten attributes of health literate health care organizations. NAM Perspectives. Discussion Paper. Washington, DC: National Academy of Medicine; 2012. https://doi.org/10.31478/201206a . Available at: https://nam.edu/perspectives-2012-tenattributes-of-health-literate-health-care-organizations/ .

Sørensen K, Levin-Zamir D, Duong TV, Okan O, Brasil VV, Nutbeam D. Building health literacy system capacity: a framework for health literate systems. Health Promot Int. 2021;36(Supplement_1):i13–23.

Download references

Acknowledgements

We thank all the study participants, the University of Akureyri Research Fund and the Akureyri Hospital Science Fund for supporting our study. Asta Sigurdardottir, a senior citizen in Akureyri and a member of our research team, is especially thanked for her valuable input and reflection.

This work was supported by the University of Akureyri Research Fund under grant number R2112 and the Akureyri Hospital Science Fund. The funders had no role in the study design, data collection and analysis or manuscript preparation.

Author information

Authors and affiliations.

Faculty of Occupational Therapy, School of Health, Business and Natural Sciences, University of Akureyri, Nordurslod 2, Akureyri, 600, Iceland

Sonja Stelly Gustafsdottir

Department of Physical Therapy, Faculty of Medicine, School of Health Sciences, University of Iceland, Reykjavik, Iceland

Sonja Stelly Gustafsdottir & Solveig A. Arnadottir

Health and Rehabilitation at the Institute of Neuroscience and Physiology, University of Gothenburg, Gothenburg, Sweden

Lena Mårtensson

Faculty of Nursing, School of Health, Business and Natural Sciences, University of Akureyri, Akureyri, Iceland

Arun K. Sigurdardottir

Akureyri Hospital, Akureyri, Iceland

You can also search for this author in PubMed   Google Scholar

Contributions

SSG collected, analysed and interpreted the data, drafted the manuscript and was the grant holder. LM contributed to analysing and interpreting the data and drafting and revising the manuscript. SAA and AKS contributed to the data interpretation and revising the manuscript. AKS was also a grant holder. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Sonja Stelly Gustafsdottir .

Ethics declarations

Ethics approval and consent to participate.

According to the Icelandic National Bioethics Committee, no further approval from the previous study (VSNb2016060007/03.01) was needed for this continued research (VSN-21–009). Information about the names and social numbers of participants and research numbers from the previous study were kept separate and accessible only to the researchers through special permission from the file holder, University of Akureyri Research Centre. Written informed consent was obtained from all participants. Participants chose pseudonyms at the beginning of each interview, which were used to ensure confidentiality and privacy, and recorded interviews were deleted after transcriptions. This study is not a registered clinical trial.

Consent for publication

Consent for publication was not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Gustafsdottir, S.S., Mårtensson, L., Sigurdardottir, A.K. et al. When great responsibility comes with limited options: experiences and needs of older community-dwelling adults regarding accessing, understanding, appraising and using health-related information. BMC Geriatr 24 , 640 (2024). https://doi.org/10.1186/s12877-024-05236-2

Download citation

Received : 03 January 2024

Accepted : 22 July 2024

Published : 31 July 2024

DOI : https://doi.org/10.1186/s12877-024-05236-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Access to health care
• Access to information
• Health promotion
• Information seeking behaviour
• Qualitative research

BMC Geriatrics

ISSN: 1471-2318

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Violence against women and children in Yemen: A mixed-methods systematic review

• Find this author on Google Scholar
• Find this author on PubMed
• Search for this author on this site
• ORCID record for Samana Shreedhar
• ORCID record for Sayali Arvind Chavan
• ORCID record for Marwah Al-Zumair
• ORCID record for Mirna Naccache
• ORCID record for Priya Shreedhar
• ORCID record for Lauren Maxwell
• For correspondence: [email protected]
• Info/History
• Supplementary material
• Preview PDF

Violence against women and children (VAWC) is a significant health and human rights issue closely tied to multiple Sustainable Development Goals. While VAWC is prevalent in all countries, the severity and incidence of VAWC increase during wars, natural disasters, economic crises, and pandemics, all of which have affected Yemen in recent years. This systematic review synthesizes evidence from qualitative and quantitative studies on the types, prevalence, perpetrators of, and risk factors for VAWC in Yemen. Before initiating the search, the protocol and search strategy were registered to PROSPERO (CRD42021237855). We systematically searched four biomedical databases and grey literature sources and used reverse snowball sampling to identify eligible studies. The 31 studies included in the analysis depicted a range of forms of VAWC, encompassing honor violence, female genital mutilation and cutting, early and very early marriage, tourist marriage, family and intimate partner violence, and gender inequities in access to food, education, and medical care. Included studies reported a high prevalence of many forms of violence, including corporal punishment in schools and intimate partner violence. We reviewed study quality and how studies addressed ethical concerns in VAWC-related research. We found that several studies did not report ethics review or interviewer training and no studies discussed safety planning or addressing the mental health needs of participants in VAWC research. This systematic review provides a much-needed synthesis of existing research on VAWC in Yemen. Since the start of the 2014 war, Yemen has become the world’s largest humanitarian crisis, with the highest rate of maternal mortality and gender inequality in the world. We only identified one study initiated after the recent war in Yemen. This deficiency represents a missed opportunity to understand how the ongoing war has reversed prior gains in reducing the prevalence of child and very early child marriage and introduced new forms of gender-based violence, including tourist marriage.

Competing Interest Statement

The authors have declared no competing interest.

Clinical Protocols

https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021237855

Funding Statement

MAZ received funding support from the Else Kroener-Fresenius-Stiftung within the Heidelberg Institute for Global Health at Universitaetsklinikum Heidelberg for one year of her work on this project (Award Number D10053008). The Heidelberg University Library supported the open-access fee for this article. The Else Kroener-Fresenius-Stiftung had no role in the study's design, conduct or reporting.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Data Availability

All data generated in this systematic review are available in the article or its appendices.

View the discussion thread.

Supplementary Material

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Citation Manager Formats

• EndNote (tagged)
• EndNote 8 (xml)
• RefWorks Tagged
• Ref Manager
• Tweet Widget
• Facebook Like
• Google Plus One

Subject Area

• Public and Global Health
• Addiction Medicine (336)
• Allergy and Immunology (658)
• Anesthesia (177)
• Cardiovascular Medicine (2573)
• Dentistry and Oral Medicine (310)
• Dermatology (218)
• Emergency Medicine (390)
• Endocrinology (including Diabetes Mellitus and Metabolic Disease) (915)
• Epidemiology (12096)
• Forensic Medicine (10)
• Gastroenterology (744)
• Genetic and Genomic Medicine (3998)
• Geriatric Medicine (376)
• Health Economics (667)
• Health Informatics (2578)
• Health Policy (993)
• Health Systems and Quality Improvement (961)
• Hematology (357)
• HIV/AIDS (826)
• Infectious Diseases (except HIV/AIDS) (13575)
• Intensive Care and Critical Care Medicine (783)
• Medical Education (396)
• Medical Ethics (107)
• Nephrology (424)
• Neurology (3764)
• Nursing (207)
• Nutrition (559)
• Obstetrics and Gynecology (719)
• Occupational and Environmental Health (689)
• Oncology (1960)
• Ophthalmology (571)
• Orthopedics (233)
• Otolaryngology (301)
• Pain Medicine (247)
• Palliative Medicine (72)
• Pathology (469)
• Pediatrics (1089)
• Pharmacology and Therapeutics (453)
• Primary Care Research (442)
• Psychiatry and Clinical Psychology (3355)
• Public and Global Health (6432)
• Radiology and Imaging (1366)
• Rehabilitation Medicine and Physical Therapy (793)
• Respiratory Medicine (859)
• Rheumatology (394)
• Sexual and Reproductive Health (401)
• Sports Medicine (336)
• Surgery (433)
• Toxicology (51)
• Transplantation (184)
• Urology (164)