research papers on health care

• Search by keyword
• Search by citation

Page 1 of 520

Navigating sexual minority identity in sport: a qualitative exploration of sexual minority student-athletes in China

Sexual minority student-athletes (SMSAs) face discrimination and identity conflicts in intercollegiate sport, impacting their participation and mental health. This study explores the perceptions of Chinese SMS...

• View Full Text

War and peace in public health education and training: a scoping review

Armed conflict and war are public health disasters. Public health action has a crucial role in conflict-related emergencies and rehabilitation but also in war prevention and peace promotion. Translating this i...

Association of social capital and adherence to a Mediterranean diet in adults living in Tehran: a cross-sectional study

Understanding the role of social capital in adherence to the Mediterranean diet can inform interventions to promote healthier eating habits in Iranian adults.

From smoke to stroke: quantifying the impact of smoking on stroke prevalence

The objective of this study is to assess the impact of smoking on stroke prevalence and to delineate the relationship between smoking-related factors and the risk of stroke, incorporating an analysis of demogr...

Influence of tuberculosis knowledge on acceptance of preventive treatment and the moderating role of tuberculosis stigma among China’s general population: cross-sectional analysis

Preventive treatment of tuberculosis infection (TBI) is considered a crucial strategy to prevent and control tuberculosis (TB). However, the acceptance and completion rates of preventive therapy for TBI are st...

Chemotherapy delays among cancer patients in Iran during COVID-19 pandemic

Following the outbreak of COVID-19, a set of restrictions, health advice, and limitations were put in place to reduce the spread of the virus. These restrictions, together with fear and anxiety of the populati...

Coverage of policies to improve antimicrobial stewardship in human medicine in low and middle income countries: results from the Global Survey of Experts on Antimicrobial Resistance

Antimicrobial resistance (AMR) constitutes a major threat to global health. While antimicrobial misuse or overuse is one of the main drivers for AMR, little is known about the extent to which antibiotic misuse...

Pre-exposure prophylaxis (PrEP) use trajectories and incidence of HIV and other sexually transmitted infections among PrEP users in Belgium: a cohort analysis of insurance claims data from 2017 to 2019

Since June 2017, oral pre-exposure prophylaxis (PrEP) has been reimbursed in Belgium for people at substantial risk of HIV. To inform the national PrEP programme, we described sociodemographic characteristics ...

Pesticide safety practice and its public health risk in African regions: systematic review and meta-analysis

Although pesticides play an integral role in food security and preventing public health from vector-borne diseases, inappropriate handling and continual use of restricted organochlorine pesticides pose short- ...

The effect of earthquake exposure on PTSD symptoms is mediated by intrusive rumination and moderated by gender: a cross-sectional study on the 2023 Kahramanmaraş earthquake survivors

Evidence suggests that natural disasters, such as earthquakes, can lead to post-traumatic stress disorder (PTSD), with women being particularly vulnerable. However, there is a dearth of knowledge about the und...

Experience and satisfaction of participants in colorectal cancer screening programs: a qualitative evidence synthesis

Experience and satisfaction of colorectal cancer screening program participants are among the key factors that determine adherence to these programs. Understanding them is crucial to ensure future participation.

Association between gestational arsenic exposure and infant physical development: a prospective cohort study

Arsenic pollution is widespread worldwide. The association between gestational arsenic exposure and adverse birth outcomes has been demonstrated in previous studies; however, few investigations have examined w...

Risk and protective factors associated with substance use among Puerto Rican youths after Hurricane María: a cross-sectional study

Identifying factors associated with post-disaster youth substance use is a crucial element of developing evidence-based prevention and intervention efforts. Hurricane María struck Puerto Rico in September of 2...

Domestic, family and sexual violence polyvictimisation and health experiences of Australian nurses, midwives and carers: a cross-sectional study

Domestic, family and sexual violence is a prevalent health and social issue. Nurses may be exposed to higher rates of this violence in their personal lives compared to the community, but little is known about ...

BMI changes among U.S. Navy and Marine Corps active-duty service members during the COVID-19 pandemic, 2019–2021

The COVID-19 pandemic disrupted the daily life and routines of Americans across the United States (U.S.), including those of our active-duty service members (ADSMs). Limited movement orders enacted during this...

RESTART: a stepped-care approach to facilitate return to work for employees with psychological distress: design of a randomized controlled trial

Common mental health problems, such as stress, anxiety and depression, are highly prevalent among workers and often lead to long-term absenteeism and work disability. Effective elements found in previous resea...

The impact of health insurance on hypertension care: a household fixed effects study in India

Hypertension is highly prevalent in India, but the proportion of patients achieving blood pressure control remains low. Efforts have been made to expand health insurance coverage nationwide with the aim of imp...

Association of serum calcium, vitamin D, and C-reactive protein with all-cause and cause-specific mortality in an osteoarthritis population in the UK: a prospective cohort study

Osteoarthritis is a prevalent musculoskeletal condition, but the role of specific serum biomarkers, such as calcium, vitamin D, and C-reactive protein (CRP), in predicting mortality among individuals with oste...

Comparison of univariate and bivariate Poisson regression methods in the analysis of determinants of female schooling and fertility in Malawi

Recent research has established existence of a correlation between women’s education and fertility, suggesting that they share similar risk factors. However, in many studies, the two variables were analysed se...

Comparing temporal changes and predictors of different types of mental health and socio-emotional wellbeing outcomes during COVID-19: an overlapping panel study of Spanish residents

Few panel studies have investigated how different types of mental health (MH) and socio-emotional wellbeing (SEW) outcomes have changed during the pandemic and if their burden has been equally distributed at t...

Cross-lagged panel analysis of the relationship between social networking sites use (SNSU) and sleep problems among university students

Sleep remains a cornerstone for sociopsychological well-being, but it is in decline, especially among today’s youth. Simultaneously, engagement with social media is escalating. Research has identified a link b...

Hospital case fatality and mortality related to Chagas disease in Brazil over two decades

To analyse hospital case fatality and mortality related to Chagas disease (CD) in Brazil, 2000–2019.

Health behavior inventory – short form: a Chinese validation study using classical test theory and item response theory

It is imperative to use a validated tool to measure and understand health behavior as it greatly impacts health status. It has been demonstrated that the Health Behavior Inventory - Short Form (HBI-SF) is vali...

Maternal mortality in Eastern Democratic Republic of Congo: a 10-year multi-zonal institutional death review

Maternal mortality (MM) remains a real scourge that hits hardest in the poorest regions of the world, particularly those affected by conflict. However, despite this worrying reality, few studies have been cond...

Health lifestyles of six Zhiguo ethnic groups in China: a latent class analysis

Zhiguo ethnic groups, commonly known as “the directly-entering-socialism ethnic groups”, represent Chinese ethnic minorities who have undergone a unique social development trajectory by transforming directly from...

Association between current cigarette prices and cessation behaviors among male adult smokers: findings from 2018 to 2020 ITC Vietnam surveys

This study evaluated the impact of the tax increase in January 2019 on changes in intention to quit and the effect of cigarette prices on quit attempts and successful quitting among male cigarette smokers in V...

Knowledge and willingness to donate kidney for transplantation among general population in Saudi Arabia

Kidney transplantation is the preferred treatment for end-stage renal disease (ESRD), offering a superior quality of life and extended survival compared to other renal replacement therapies. As the number of E...

Co-producing an intervention to prevent mental health problems in children and young people in contact with child welfare services

Children and young people (CYP) in contact with child welfare services are at high risk of developing mental health problems. There is a paucity of evidenced-based preventative interventions provided to this p...

Community Control of Hypertension and Diabetes (CoCo-HD) program in the Indian states of Kerala and Tamil Nadu: a study protocol for a type 3 hybrid trial

India grapples with a formidable health challenge, with an estimated 315 million adults afflicted with hypertension and 100 million living with diabetes mellitus. Alarming statistics reveal rates for poor trea...

Summer temperature and emergency room visits due to urinary tract infection in South Korea: a national time-stratified case-crossover study

Although urinary tract infection (UTI) is a common and severe public health concern, and there are clear biological mechanisms between UTI and hot temperatures, few studies have addressed the association betwe...

Trends in transport injuries burden and risk factors among children under 14 years old in China: 1990–2019

Transport injuries (TI) remains one of leading causes of death in children in China. This study aimed to analyze the temporal trend of disease burden and associated risk factors of TI among children aged 0–14 ...

Facilitators and barriers to implementing successful exclusion among children with shiga toxin-producing Escherichia coli : a qualitative analysis of public health case management records

Shiga toxin-producing Escherichia coli (STEC) infections are a significant public health concern as they can cause serious illness and outbreaks. In England, STEC incidence is highest among children and guidance ...

Measuring preparedness to infectious diseases among people exposed to climate disasters in Cabo Delgado, Mozambique: a cross-sectional study

Climate change is contributing to increase the frequency and severity of climate disasters in Mozambique, leading, since 2019, to extensive damage to infrastructure and displacement 1.3 million people. Aim of ...

Knowledge, attitudes, and practices of government workers on zoonotic parasites in pet animals: a study in Central Gondar Zone, Ethiopia

Parasitic zoonoses are infections transmitted between vertebrate animals and humans, posing serious public health risks and significant economic consequences. The study aimed at assessing government workers' k...

Association of the visceral fat metabolic score with osteoarthritis risk: a cross-sectional study from NHANES 2009–2018

Currently, obesity has been recognized to be an independent risk factor for osteoarthritis (OA), and the Metabolic Score for Visceral Fat (METS-VF) has been suggested to be potentially more accurate than body ...

Assessment of national reimbursement anticancer medications availability in China: based on a medical institution perspective

High clinical value national reimbursement anticancer medications (NRAMs) are pivotal treatments for patients with cancer. However, the availability of NRAMs in medical institutions is unknown. This study aime...

The relationship between lifestyle habits and obesity among students in the Eastern province of Saudi Arabia: using the Arab Teens Lifestyle (ATLS) questionnaire

The Arab Teens Lifestyle (ATLS) questionnaire was an initiative to assess the lifestyle habits influencing obesity rates in the Middle East and North Africa (MENA) region, including physical activity (PA) patt...

The acceptability and effectiveness of artificial intelligence-based chatbot for hypertensive patients in community: protocol for a mixed-methods study

Chatbots can provide immediate assistance tailored to patients’ needs, making them suitable for sustained accompanying interventions. Nevertheless, there is currently no evidence regarding their acceptability ...

Long COVID impacts: the voices and views of diverse Black and Latinx residents in Massachusetts

To understand how Long COVID is impacting the health and social conditions of the Black and Latinx communities.

Sex differences in physical activity of US children at age 13 months: child and Mother Physical Activity Study (CAMPAS)

Lower physical activity (PA) has been observed in females compared to males among preschool-aged and older children. However, the timing of when the sex gap emerges is unclear. The purpose of this study was to...

Oral health knowledge, attitudes, and practices of people living with mental illness: a mixed-methods systematic review

People living with mental illness experience poorer oral health outcomes compared to the general population, yet little is known about their oral health knowledge, attitudes, and practices. The aim of this mix...

The effect of long-term care insurance on women’s health outcomes in China: evidence from a disability inclusion action

Long-term care insurance (LTCI) in China provides financial and care security for persons with disabilities and includes caregivers in the paid labour workforce. However, it is unclear how the LTCI affects hea...

Process evaluation of an mHealth-based school education program to reduce salt intake scaling up in China (EduSaltS): a mixed methods study using the RE-AIM framework

An mHealth-based school health education platform (EduSaltS) was promoted in real-world China to reduce salt intake among children and their families. This progress evaluation explores its implementation proce...

Association between dietary patterns and depression symptoms among adults with or without diabetes in Qatar: a population-based study

Diabetes is a major public health problem in Qatar and is associated with an increased risk of depression. However, no study has been conducted in Qatar on the relationship between dietary patterns and depress...

Metabolic syndrome and cardiovascular disease among adult cancer patients: results from NHANES 2007–2018

Metabolic syndrome (MetS) is a risk factor for cardiovascular disease (CVD), and CVD is a major challenge for cancer patients. This study aimed to investigate the prevalence and association of MetS and CVD amo...

Patient perspective on the elimination mother-to-child transmission of HIV, syphilis, and hepatitis B in Bali, Indonesia: a qualitative study

This study aimed to explore the facilitators and barriers to the elimination of human immunodeficiency virus (HIV), syphilis, and hepatitis B transmission based on the perspectives of mothers living with HIV, ...

Work-related stress among nurses: a comparative cross-sectional study of two government hospitals in Ghana

Occupational stress is increasing globally and affecting all workplaces and countries’ health professionals. Despite its significant impact on healthcare delivery, limited studies have actually compared the pr...

Awareness and knowledge of cardiovascular disease and their associations with the lifestyles of 8-16-year-old girls: a perspective from Saudi Arabia

There has been an increased focus on children as a target for primordial prevention of cardiovascular disease (CVD). In particular, girls should receive attention to combat the burden of CVD in women, because ...

Multicentre case–control study of pneumococcal infections among children with pneumonia in Peninsular Malaysia (MY-Pneumo): a study protocol

S. pneumoniae (SPN) is the most common cause of pneumonia. The disease can be effectively prevented through immunisation. Since December 2020, the Malaysian Government has included the 10-valent pneumococcal conj...

Building running-friendly cities: effects of streetscapes on running using 9.73 million fitness tracker data in Shanghai, China

The association between built environment and physical activity has been recognized. However, how and to what extent microscale streetscapes are related to running activity remains underexplored, partly due to...

Important information

Editorial board

For authors

For editorial board members

For reviewers

• Manuscript editing services

Annual Journal Metrics

Citation Impact 2023 Journal Impact Factor: 3.5 5-year Journal Impact Factor: 3.9 Source Normalized Impact per Paper (SNIP): 1.386 SCImago Journal Rank (SJR): 1.253 Speed 2023 Submission to first editorial decision (median days): 7 Submission to acceptance (median days): 173 Usage 2023 Downloads: 24,332,405 Altmetric mentions: 24,308

• More about our metrics

Peer-review Terminology

The following summary describes the peer review process for this journal:

Identity transparency: Single anonymized

Reviewer interacts with: Editor

Review information published: Review reports. Reviewer Identities reviewer opt in. Author/reviewer communication

More information is available here

• Follow us on Twitter

BMC Public Health

ISSN: 1471-2458

• General enquiries: [email protected]

Loading metrics

Open Access

Peer-reviewed

Research Article

Assessing the impact of healthcare research: A systematic review of methodological frameworks

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Validation, Writing – original draft, Writing – review & editing

Affiliation Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom

Roles Conceptualization, Formal analysis, Funding acquisition, Methodology, Project administration, Supervision, Validation, Writing – review & editing

* E-mail: [email protected]

Roles Data curation, Formal analysis, Methodology, Validation, Writing – review & editing

Roles Formal analysis, Methodology, Supervision, Validation, Writing – review & editing

• Samantha Cruz Rivera, 
• Derek G. Kyte, 
• Olalekan Lee Aiyegbusi, 
• Thomas J. Keeley, 
• Melanie J. Calvert

• Published: August 9, 2017
• https://doi.org/10.1371/journal.pmed.1002370
• Reader Comments

Increasingly, researchers need to demonstrate the impact of their research to their sponsors, funders, and fellow academics. However, the most appropriate way of measuring the impact of healthcare research is subject to debate. We aimed to identify the existing methodological frameworks used to measure healthcare research impact and to summarise the common themes and metrics in an impact matrix.

Methods and findings

Two independent investigators systematically searched the Medical Literature Analysis and Retrieval System Online (MEDLINE), the Excerpta Medica Database (EMBASE), the Cumulative Index to Nursing and Allied Health Literature (CINAHL+), the Health Management Information Consortium, and the Journal of Research Evaluation from inception until May 2017 for publications that presented a methodological framework for research impact. We then summarised the common concepts and themes across methodological frameworks and identified the metrics used to evaluate differing forms of impact. Twenty-four unique methodological frameworks were identified, addressing 5 broad categories of impact: (1) ‘primary research-related impact’, (2) ‘influence on policy making’, (3) ‘health and health systems impact’, (4) ‘health-related and societal impact’, and (5) ‘broader economic impact’. These categories were subdivided into 16 common impact subgroups. Authors of the included publications proposed 80 different metrics aimed at measuring impact in these areas. The main limitation of the study was the potential exclusion of relevant articles, as a consequence of the poor indexing of the databases searched.

Conclusions

The measurement of research impact is an essential exercise to help direct the allocation of limited research resources, to maximise research benefit, and to help minimise research waste. This review provides a collective summary of existing methodological frameworks for research impact, which funders may use to inform the measurement of research impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their research.

Author summary

Why was this study done.

• There is a growing interest in demonstrating the impact of research in order to minimise research waste, allocate resources efficiently, and maximise the benefit of research. However, there is no consensus on which is the most appropriate tool to measure the impact of research.
• To our knowledge, this review is the first to synthesise existing methodological frameworks for healthcare research impact, and the associated impact metrics by which various authors have proposed impact should be measured, into a unified matrix.

What did the researchers do and find?

• We conducted a systematic review identifying 24 existing methodological research impact frameworks.
• We scrutinised the sample, identifying and summarising 5 proposed impact categories, 16 impact subcategories, and over 80 metrics into an impact matrix and methodological framework.

What do these findings mean?

• This simplified consolidated methodological framework will help researchers to understand how a research study may give rise to differing forms of impact, as well as in what ways and at which time points these potential impacts might be measured.
• Incorporating these insights into the design of a study could enhance impact, optimizing the use of research resources.

Citation: Cruz Rivera S, Kyte DG, Aiyegbusi OL, Keeley TJ, Calvert MJ (2017) Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS Med 14(8): e1002370. https://doi.org/10.1371/journal.pmed.1002370

Academic Editor: Mike Clarke, Queens University Belfast, UNITED KINGDOM

Received: February 28, 2017; Accepted: July 7, 2017; Published: August 9, 2017

Copyright: © 2017 Cruz Rivera et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and supporting files.

Funding: Funding was received from Consejo Nacional de Ciencia y Tecnología (CONACYT). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript ( http://www.conacyt.mx/ ).

Competing interests: I have read the journal's policy and the authors of this manuscript have the following competing interests: MJC has received consultancy fees from Astellas and Ferring pharma and travel fees from the European Society of Cardiology outside the submitted work. TJK is in full-time paid employment for PAREXEL International.

Abbreviations: AIHS, Alberta Innovates—Health Solutions; CAHS, Canadian Academy of Health Sciences; CIHR, Canadian Institutes of Health Research; CINAHL+, Cumulative Index to Nursing and Allied Health Literature; EMBASE, Excerpta Medica Database; ERA, Excellence in Research for Australia; HEFCE, Higher Education Funding Council for England; HMIC, Health Management Information Consortium; HTA, Health Technology Assessment; IOM, Impact Oriented Monitoring; MDG, Millennium Development Goal; NHS, National Health Service; MEDLINE, Medical Literature Analysis and Retrieval System Online; PHC RIS, Primary Health Care Research & Information Service; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; PROM, patient-reported outcome measures; QALY, quality-adjusted life year; R&D, research and development; RAE, Research Assessment Exercise; REF, Research Excellence Framework; RIF, Research Impact Framework; RQF, Research Quality Framework; SDG, Sustainable Development Goal; SIAMPI, Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions between science and society

Introduction

In 2010, approximately US$240 billion was invested in healthcare research worldwide [ 1 ]. Such research is utilised by policy makers, healthcare providers, and clinicians to make important evidence-based decisions aimed at maximising patient benefit, whilst ensuring that limited healthcare resources are used as efficiently as possible to facilitate effective and sustainable service delivery. It is therefore essential that this research is of high quality and that it is impactful—i.e., it delivers demonstrable benefits to society and the wider economy whilst minimising research waste [ 1 , 2 ]. Research impact can be defined as ‘any identifiable ‘benefit to, or positive influence on the economy, society, public policy or services, health, the environment, quality of life or academia’ (p. 26) [ 3 ].

There are many purported benefits associated with the measurement of research impact, including the ability to (1) assess the quality of the research and its subsequent benefits to society; (2) inform and influence optimal policy and funding allocation; (3) demonstrate accountability, the value of research in terms of efficiency and effectiveness to the government, stakeholders, and society; and (4) maximise impact through better understanding the concept and pathways to impact [ 4 – 7 ].

Measuring and monitoring the impact of healthcare research has become increasingly common in the United Kingdom [ 5 ], Australia [ 5 ], and Canada [ 8 ], as governments, organisations, and higher education institutions seek a framework to allocate funds to projects that are more likely to bring the most benefit to society and the economy [ 5 ]. For example, in the UK, the 2014 Research Excellence Framework (REF) has recently been used to assess the quality and impact of research in higher education institutions, through the assessment of impact cases studies and selected qualitative impact metrics [ 9 ]. This is the first initiative to allocate research funding based on the economic, societal, and cultural impact of research, although it should be noted that research impact only drives a proportion of this allocation (approximately 20%) [ 9 ].

In the UK REF, the measurement of research impact is seen as increasingly important. However, the impact element of the REF has been criticised in some quarters [ 10 , 11 ]. Critics deride the fact that REF impact is determined in a relatively simplistic way, utilising researcher-generated case studies, which commonly attempt to link a particular research outcome to an associated policy or health improvement despite the fact that the wider literature highlights great diversity in the way research impact may be demonstrated [ 12 , 13 ]. This led to the current debate about the optimal method of measuring impact in the future REF [ 10 , 14 ]. The Stern review suggested that research impact should not only focus on socioeconomic impact but should also include impact on government policy, public engagement, academic impacts outside the field, and teaching to showcase interdisciplinary collaborative impact [ 10 , 11 ]. The Higher Education Funding Council for England (HEFCE) has recently set out the proposals for the REF 2021 exercise, confirming that the measurement of such impact will continue to form an important part of the process [ 15 ].

With increasing pressure for healthcare research to lead to demonstrable health, economic, and societal impact, there is a need for researchers to understand existing methodological impact frameworks and the means by which impact may be quantified (i.e., impact metrics; see Box 1 , 'Definitions’) to better inform research activities and funding decisions. From a researcher’s perspective, understanding the optimal pathways to impact can help inform study design aimed at maximising the impact of the project. At the same time, funders need to understand which aspects of impact they should focus on when allocating awards so they can make the most of their investment and bring the greatest benefit to patients and society [ 2 , 4 , 5 , 16 , 17 ].

Box 1. Definitions

• Research impact: ‘any identifiable benefit to, or positive influence on, the economy, society, public policy or services, health, the environment, quality of life, or academia’ (p. 26) [ 3 ].
• Methodological framework: ‘a body of methods, rules and postulates employed by a particular procedure or set of procedures (i.e., framework characteristics and development)’ [ 18 ].
• Pathway: ‘a way of achieving a specified result; a course of action’ [ 19 ].
• Quantitative metrics: ‘a system or standard of [quantitative] measurement’ [ 20 ].
• Narrative metrics: ‘a spoken or written account of connected events; a story’ [ 21 ].

Whilst previous researchers have summarised existing methodological frameworks and impact case studies [ 4 , 22 – 27 ], they have not summarised the metrics for use by researchers, funders, and policy makers. The aim of this review was therefore to (1) identify the methodological frameworks used to measure healthcare research impact using systematic methods, (2) summarise common impact themes and metrics in an impact matrix, and (3) provide a simplified consolidated resource for use by funders, researchers, and policy makers.

Search strategy and selection criteria

Initially, a search strategy was developed to identify the available literature regarding the different methods to measure research impact. The following keywords: ‘Impact’, ‘Framework’, and ‘Research’, and their synonyms, were used during the search of the Medical Literature Analysis and Retrieval System Online (MEDLINE; Ovid) database, the Excerpta Medica Database (EMBASE), the Health Management Information Consortium (HMIC) database, and the Cumulative Index to Nursing and Allied Health Literature (CINAHL+) database (inception to May 2017; see S1 Appendix for the full search strategy). Additionally, the nonindexed Journal of Research Evaluation was hand searched during the same timeframe using the keyword ‘Impact’. Other relevant articles were identified through 3 Internet search engines (Google, Google Scholar, and Google Images) using the keywords ‘Impact’, ‘Framework’, and ‘Research’, with the first 50 results screened. Google Images was searched because different methodological frameworks are summarised in a single image and can easily be identified through this search engine. Finally, additional publications were sought through communication with experts.

Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (see S1 PRISMA Checklist ), 2 independent investigators systematically screened for publications describing, evaluating, or utilising a methodological research impact framework within the context of healthcare research [ 28 ]. Papers were eligible if they included full or partial methodological frameworks or pathways to research impact; both primary research and systematic reviews fitting these criteria were included. We included any methodological framework identified (original or modified versions) at the point of first occurrence. In addition, methodological frameworks were included if they were applicable to the healthcare discipline with no need of modification within their structure. We defined ‘methodological framework’ as ‘a body of methods, rules and postulates employed by a particular procedure or set of procedures (i.e., framework characteristics and development)’ [ 18 ], whereas we defined ‘pathway’ as ‘a way of achieving a specified result; a course of action’ [ 19 ]. Studies were excluded if they presented an existing (unmodified) methodological framework previously available elsewhere, did not explicitly describe a methodological framework but rather focused on a single metric (e.g., bibliometric analysis), focused on the impact or effectiveness of interventions rather than that of the research, or presented case study data only. There were no language restrictions.

Data screening

Records were downloaded into Endnote (version X7.3.1), and duplicates were removed. Two independent investigators (SCR and OLA) conducted all screening following a pilot aimed at refining the process. The records were screened by title and abstract before full-text articles of potentially eligible publications were retrieved for evaluation. A full-text screening identified the publications included for data extraction. Discrepancies were resolved through discussion, with the involvement of a third reviewer (MJC, DGK, and TJK) when necessary.

Data extraction and analysis

Data extraction occurred after the final selection of included articles. SCR and OLA independently extracted details of impact methodological frameworks, the country of origin, and the year of publication, as well as the source, the framework description, and the methodology used to develop the framework. Information regarding the methodology used to develop each methodological framework was also extracted from framework webpages where available. Investigators also extracted details regarding each framework’s impact categories and subgroups, along with their proposed time to impact (‘short-term’, ‘mid-term’, or ‘long-term’) and the details of any metrics that had been proposed to measure impact, which are depicted in an impact matrix. The structure of the matrix was informed by the work of M. Buxton and S. Hanney [ 2 ], P. Buykx et al. [ 5 ], S. Kuruvila et al. [ 29 ], and A. Weiss [ 30 ], with the intention of mapping metrics presented in previous methodological frameworks in a concise way. A consensus meeting with MJC, DGK, and TJK was held to solve disagreements and finalise the data extraction process.

Included studies

Our original search strategy identified 359 citations from MEDLINE (Ovid), EMBASE, CINAHL+, HMIC, and the Journal of Research Evaluation, and 101 citations were returned using other sources (Google, Google Images, Google Scholar, and expert communication) (see Fig 1 ) [ 28 ]. In total, we retrieved 54 full-text articles for review. At this stage, 39 articles were excluded, as they did not propose new or modified methodological frameworks. An additional 15 articles were included following the backward and forward citation method. A total of 31 relevant articles were included in the final analysis, of which 24 were articles presenting unique frameworks and the remaining 7 were systematic reviews [ 4 , 22 – 27 ]. The search strategy was rerun on 15 May 2017. A further 19 publications were screened, and 2 were taken forward to full-text screening but were ineligible for inclusion.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pmed.1002370.g001

Methodological framework characteristics

The characteristics of the 24 included methodological frameworks are summarised in Table 1 , 'Methodological framework characteristics’. Fourteen publications proposed academic-orientated frameworks, which focused on measuring academic, societal, economic, and cultural impact using narrative and quantitative metrics [ 2 , 3 , 5 , 8 , 29 , 31 – 39 ]. Five publications focused on assessing the impact of research by focusing on the interaction process between stakeholders and researchers (‘productive interactions’), which is a requirement to achieve research impact. This approach tries to address the issue of attributing research impact to metrics [ 7 , 40 – 43 ]. Two frameworks focused on the importance of partnerships between researchers and policy makers, as a core element to accomplish research impact [ 44 , 45 ]. An additional 2 frameworks focused on evaluating the pathways to impact, i.e., linking processes between research and impact [ 30 , 46 ]. One framework assessed the ability of health technology to influence efficiency of healthcare systems [ 47 ]. Eight frameworks were developed in the UK [ 2 , 3 , 29 , 37 , 39 , 42 , 43 , 45 ], 6 in Canada [ 8 , 33 , 34 , 44 , 46 , 47 ], 4 in Australia [ 5 , 31 , 35 , 38 ], 3 in the Netherlands [ 7 , 40 , 41 ], and 2 in the United States [ 30 , 36 ], with 1 model developed with input from various countries [ 32 ].

https://doi.org/10.1371/journal.pmed.1002370.t001

Methodological framework development

The included methodological frameworks varied in their development process, but there were some common approaches employed. Most included a literature review [ 2 , 5 , 7 , 8 , 31 , 33 , 36 , 37 , 40 – 46 ], although none of them used a recognised systematic method. Most also consulted with various stakeholders [ 3 , 8 , 29 , 31 , 33 , 35 – 38 , 43 , 44 , 46 , 47 ] but used differing methods to incorporate their views, including quantitative surveys [ 32 , 35 , 43 , 46 ], face-to-face interviews [ 7 , 29 , 33 , 35 , 37 , 42 , 43 ], telephone interviews [ 31 , 46 ], consultation [ 3 , 7 , 36 ], and focus groups [ 39 , 43 ]. A range of stakeholder groups were approached across the sample, including principal investigators [ 7 , 29 , 43 ], research end users [ 7 , 42 , 43 ], academics [ 3 , 8 , 39 , 40 , 43 , 46 ], award holders [ 43 ], experts [ 33 , 38 , 39 ], sponsors [ 33 , 39 ], project coordinators [ 32 , 42 ], and chief investigators [ 31 , 35 ]. However, some authors failed to identify the stakeholders involved in the development of their frameworks [ 2 , 5 , 34 , 41 , 45 ], making it difficult to assess their appropriateness. In addition, only 4 of the included papers reported using formal analytic methods to interpret stakeholder responses. These included the Canadian Academy of Health Sciences framework, which used conceptual cluster analysis [ 33 ]. The Research Contribution [ 42 ], Research Impact [ 29 ], and Primary Health Care & Information Service [ 31 ] used a thematic analysis approach. Finally, some authors went on to pilot their framework, which shaped refinements on the methodological frameworks until approval. Methods used to pilot the frameworks included a case study approach [ 2 , 3 , 30 , 32 , 33 , 36 , 40 , 42 , 44 , 45 ], contrasting results against available literature [ 29 ], the use of stakeholders’ feedback [ 7 ], and assessment tools [ 35 , 46 ].

Major impact categories

1. primary research-related impact..

A number of methodological frameworks advocated the evaluation of ‘research-related impact’. This encompassed content related to the generation of new knowledge, knowledge dissemination, capacity building, training, leadership, and the development of research networks. These outcomes were considered the direct or primary impacts of a research project, as these are often the first evidenced returns [ 30 , 62 ].

A number of subgroups were identified within this category, with frameworks supporting the collection of impact data across the following constructs: ‘research and innovation outcomes’; ‘dissemination and knowledge transfer’; ‘capacity building, training, and leadership’; and ‘academic collaborations, research networks, and data sharing’.

1 . 1 . Research and innovation outcomes . Twenty of the 24 frameworks advocated the evaluation of ‘research and innovation outcomes’ [ 2 , 3 , 5 , 7 , 8 , 29 – 39 , 41 , 43 , 44 , 46 ]. This subgroup included the following metrics: number of publications; number of peer-reviewed articles (including journal impact factor); citation rates; requests for reprints, number of reviews, and meta-analysis; and new or changes in existing products (interventions or technology), patents, and research. Additionally, some frameworks also sought to gather information regarding ‘methods/methodological contributions’. These advocated the collection of systematic reviews and appraisals in order to identify gaps in knowledge and determine whether the knowledge generated had been assessed before being put into practice [ 29 ].

1 . 2 . Dissemination and knowledge transfer . Nineteen of the 24 frameworks advocated the assessment of ‘dissemination and knowledge transfer’ [ 2 , 3 , 5 , 7 , 29 – 32 , 34 – 43 , 46 ]. This comprised collection of the following information: number of conferences, seminars, workshops, and presentations; teaching output (i.e., number of lectures given to disseminate the research findings); number of reads for published articles; article download rate and number of journal webpage visits; and citations rates in nonjournal media such as newspapers and mass and social media (i.e., Twitter and blogs). Furthermore, this impact subgroup considered the measurement of research uptake and translatability and the adoption of research findings in technological and clinical applications and by different fields. These can be measured through patents, clinical trials, and partnerships between industry and business, government and nongovernmental organisations, and university research units and researchers [ 29 ].

1 . 3 . Capacity building , training , and leadership . Fourteen of 24 frameworks suggested the evaluation of ‘capacity building, training, and leadership’ [ 2 , 3 , 5 , 8 , 29 , 31 – 35 , 39 – 41 , 43 ]. This involved collecting information regarding the number of doctoral and postdoctoral studentships (including those generated as a result of the research findings and those appointed to conduct the research), as well as the number of researchers and research-related staff involved in the research projects. In addition, authors advocated the collection of ‘leadership’ metrics, including the number of research projects managed and coordinated and the membership of boards and funding bodies, journal editorial boards, and advisory committees [ 29 ]. Additional metrics in this category included public recognition (number of fellowships and awards for significant research achievements), academic career advancement, and subsequent grants received. Lastly, the impact metric ‘research system management’ comprised the collection of information that can lead to preserving the health of the population, such as modifying research priorities, resource allocation strategies, and linking health research to other disciplines to maximise benefits [ 29 ].

1 . 4 . Academic collaborations , research networks , and data sharing . Lastly, 10 of the 24 frameworks advocated the collection of impact data regarding ‘academic collaborations (internal and external collaborations to complete a research project), research networks, and data sharing’ [ 2 , 3 , 5 , 7 , 29 , 34 , 37 , 39 , 41 , 43 ].

2. Influence on policy making.

Methodological frameworks addressing this major impact category focused on measurable improvements within a given knowledge base and on interactions between academics and policy makers, which may influence policy-making development and implementation. The returns generated in this impact category are generally considered as intermediate or midterm (1 to 3 years). These represent an important interim stage in the process towards the final expected impacts, such as quantifiable health improvements and economic benefits, without which policy change may not occur [ 30 , 62 ]. The following impact subgroups were identified within this category: ‘type and nature of policy impact’, ‘level of policy making’, and ‘policy networks’.

2 . 1 . Type and nature of policy impact . The most common impact subgroup, mentioned in 18 of the 24 frameworks, was ‘type and nature of policy impact’ [ 2 , 7 , 29 – 38 , 41 – 43 , 45 – 47 ]. Methodological frameworks addressing this subgroup stressed the importance of collecting information regarding the influence of research on policy (i.e., changes in practice or terminology). For instance, a project looking at trafficked adolescents and women (2003) influenced the WHO guidelines (2003) on ethics regarding this particular group [ 17 , 21 , 63 ].

2 . 2 . Level of policy impact . Thirteen of 24 frameworks addressed aspects surrounding the need to record the ‘level of policy impact’ (international, national, or local) and the organisations within a level that were influenced (local policy makers, clinical commissioning groups, and health and wellbeing trusts) [ 2 , 5 , 8 , 29 , 31 , 34 , 38 , 41 , 43 – 47 ]. Authors considered it important to measure the ‘level of policy impact’ to provide evidence of collaboration, coordination, and efficiency within health organisations and between researchers and health organisations [ 29 , 31 ].

2 . 3 . Policy networks . Five methodological frameworks highlighted the need to collect information regarding collaborative research with industry and staff movement between academia and industry [ 5 , 7 , 29 , 41 , 43 ]. A policy network emphasises the relationship between policy communities, researchers, and policy makers. This relationship can influence and lead to incremental changes in policy processes [ 62 ].

3. Health and health systems impact.

A number of methodological frameworks advocated the measurement of impacts on health and healthcare systems across the following impact subgroups: ‘quality of care and service delivering’, ‘evidence-based practice’, ‘improved information and health information management’, ‘cost containment and effectiveness’, ‘resource allocation’, and ‘health workforce’.

3 . 1 . Quality of care and service delivery . Twelve of the 24 frameworks highlighted the importance of evaluating ‘quality of care and service delivery’ [ 2 , 5 , 8 , 29 – 31 , 33 – 36 , 41 , 47 ]. There were a number of suggested metrics that could be potentially used for this purpose, including health outcomes such as quality-adjusted life years (QALYs), patient-reported outcome measures (PROMs), patient satisfaction and experience surveys, and qualitative data on waiting times and service accessibility.

3 . 2 . Evidence-based practice . ‘Evidence-based practice’, mentioned in 5 of the 24 frameworks, refers to making changes in clinical diagnosis, clinical practice, treatment decisions, or decision making based on research evidence [ 5 , 8 , 29 , 31 , 33 ]. The suggested metrics to demonstrate evidence-based practice were adoption of health technologies and research outcomes to improve the healthcare systems and inform policies and guidelines [ 29 ].

3 . 3 . Improved information and health information management . This impact subcategory, mentioned in 5 of the 24 frameworks, refers to the influence of research on the provision of health services and management of the health system to prevent additional costs [ 5 , 29 , 33 , 34 , 38 ]. Methodological frameworks advocated the collection of health system financial, nonfinancial (i.e., transport and sociopolitical implications), and insurance information in order to determine constraints within a health system.

3 . 4 . Cost containment and cost-effectiveness . Six of the 24 frameworks advocated the subcategory ‘cost containment and cost-effectiveness’ [ 2 , 5 , 8 , 17 , 33 , 36 ]. ‘Cost containment’ comprised the collection of information regarding how research has influenced the provision and management of health services and its implication in healthcare resource allocation and use [ 29 ]. ‘Cost-effectiveness’ refers to information concerning economic evaluations to assess improvements in effectiveness and health outcomes—for instance, the cost-effectiveness (cost and health outcome benefits) assessment of introducing a new health technology to replace an older one [ 29 , 31 , 64 ].

3 . 5 . Resource allocation . ‘Resource allocation’, mentioned in 6frameworks, can be measured through 2 impact metrics: new funding attributed to the intervention in question and equity while allocating resources, such as improved allocation of resources at an area level; better targeting, accessibility, and utilisation; and coverage of health services [ 2 , 5 , 29 , 31 , 45 , 47 ]. The allocation of resources and targeting can be measured through health services research reports, with the utilisation of health services measured by the probability of providing an intervention when needed, the probability of requiring it again in the future, and the probability of receiving an intervention based on previous experience [ 29 , 31 ].

3 . 6 . Health workforce . Lastly, ‘health workforce’, present in 3 methodological frameworks, refers to the reduction in the days of work lost because of a particular illness [ 2 , 5 , 31 ].

4. Health-related and societal impact.

Three subgroups were included in this category: ‘health literacy’; ‘health knowledge, attitudes, and behaviours’; and ‘improved social equity, inclusion, or cohesion’.

4 . 1 . Health knowledge , attitudes , and behaviours . Eight of the 24 frameworks suggested the assessment of ‘health knowledge, attitudes, behaviours, and outcomes’, which could be measured through the evaluation of levels of public engagement with science and research (e.g., National Health Service (NHS) Choices end-user visit rate) or by using focus groups to analyse changes in knowledge, attitudes, and behaviour among society [ 2 , 5 , 29 , 33 – 35 , 38 , 43 ].

4 . 2 . Improved equity , inclusion , or cohesion and human rights . Other methodological frameworks, 4 of the 24, suggested capturing improvements in equity, inclusion, or cohesion and human rights. Authors suggested these could be using a resource like the United Nations Millennium Development Goals (MDGs) (superseded by Sustainable Development Goals [SDGs] in 2015) and human rights [ 29 , 33 , 34 , 38 ]. For instance, a cluster-randomised controlled trial in Nepal, which had female participants, has demonstrated the reduction of neonatal mortality through the introduction of maternity health care, distribution of delivery kits, and home visits. This illustrates how research can target vulnerable and disadvantaged groups. Additionally, this research has been introduced by the World Health Organisation to achieve the MDG ‘improve maternal health’ [ 16 , 29 , 65 ].

4 . 3 . Health literacy . Some methodological frameworks, 3 of the 24, focused on tracking changes in the ability of patients to make informed healthcare decisions, reduce health risks, and improve quality of life, which were demonstrably linked to a particular programme of research [ 5 , 29 , 43 ]. For example, a systematic review showed that when HIV health literacy/knowledge is spread among people living with the condition, antiretroviral adherence and quality of life improve [ 66 ].

5. Broader economic impacts.

Some methodological frameworks, 9 of 24, included aspects related to the broader economic impacts of health research—for example, the economic benefits emerging from the commercialisation of research outputs [ 2 , 5 , 29 , 31 , 33 , 35 , 36 , 38 , 67 ]. Suggested metrics included the amount of funding for research and development (R&D) that was competitively awarded by the NHS, medical charities, and overseas companies. Additional metrics were income from intellectual property, spillover effects (any secondary benefit gained as a repercussion of investing directly in a primary activity, i.e., the social and economic returns of investing on R&D) [ 33 ], patents granted, licences awarded and brought to the market, the development and sales of spinout companies, research contracts, and income from industry.

The benefits contained within the categories ‘health and health systems impact’, ‘health-related and societal impact’, and ‘broader economic impacts’ are considered the expected and final returns of the resources allocated in healthcare research [ 30 , 62 ]. These benefits commonly arise in the long term, beyond 5 years according to some authors, but there was a recognition that this could differ depending on the project and its associated research area [ 4 ].

Data synthesis

Five major impact categories were identified across the 24 included methodological frameworks: (1) ‘primary research-related impact’, (2) ‘influence on policy making’, (3) ‘health and health systems impact’, (4) ‘health-related and societal impact’, and (5) ‘broader economic impact’. These major impact categories were further subdivided into 16 impact subgroups. The included publications proposed 80 different metrics to measure research impact. This impact typology synthesis is depicted in ‘the impact matrix’ ( Fig 2 and Fig 3 ).

CIHR, Canadian Institutes of Health Research; HTA, Health Technology Assessment; PHC RIS, Primary Health Care Research & Information Service; RAE, Research Assessment Exercise; RQF, Research Quality Framework.

https://doi.org/10.1371/journal.pmed.1002370.g002

AIHS, Alberta Innovates—Health Solutions; CAHS, Canadian Institutes of Health Research; IOM, Impact Oriented Monitoring; REF, Research Excellence Framework; SIAMPI, Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions between science and society.

https://doi.org/10.1371/journal.pmed.1002370.g003

Commonality and differences across frameworks

The ‘Research Impact Framework’ and the ‘Health Services Research Impact Framework’ were the models that encompassed the largest number of the metrics extracted. The most dominant methodological framework was the Payback Framework; 7 other methodological framework models used the Payback Framework as a starting point for development [ 8 , 29 , 31 – 35 ]. Additional methodological frameworks that were commonly incorporated into other tools included the CIHR framework, the CAHS model, the AIHS framework, and the Exchange model [ 8 , 33 , 34 , 44 ]. The capture of ‘research-related impact’ was the most widely advocated concept across methodological frameworks, illustrating the importance with which primary short-term impact outcomes were viewed by the included papers. Thus, measurement of impact via number of publications, citations, and peer-reviewed articles was the most common. ‘Influence on policy making’ was the predominant midterm impact category, specifically the subgroup ‘type and nature of policy impact’, in which frameworks advocated the measurement of (i) changes to legislation, regulations, and government policy; (ii) influence and involvement in decision-making processes; and (iii) changes to clinical or healthcare training, practice, or guidelines. Within more long-term impact measurement, the evaluations of changes in the ‘quality of care and service delivery’ were commonly advocated.

In light of the commonalities and differences among the methodological frameworks, the ‘pathways to research impact’ diagram ( Fig 4 ) was developed to provide researchers, funders, and policy makers a more comprehensive and exhaustive way to measure healthcare research impact. The diagram has the advantage of assorting all the impact metrics proposed by previous frameworks and grouping them into different impact subgroups and categories. Prospectively, this global picture will help researchers, funders, and policy makers plan strategies to achieve multiple pathways to impact before carrying the research out. The analysis of the data extraction and construction of the impact matrix led to the development of the ‘pathways to research impact’ diagram ( Fig 4 ). The diagram aims to provide an exhaustive and comprehensive way of tracing research impact by combining all the impact metrics presented by the different 24 frameworks, grouping those metrics into different impact subgroups, and grouping these into broader impact categories.

NHS, National Health Service; PROM, patient-reported outcome measure; QALY, quality-adjusted life year; R&D, research and development.

https://doi.org/10.1371/journal.pmed.1002370.g004

This review has summarised existing methodological impact frameworks together for the first time using systematic methods ( Fig 4 ). It allows researchers and funders to consider pathways to impact at the design stage of a study and to understand the elements and metrics that need to be considered to facilitate prospective assessment of impact. Users do not necessarily need to cover all the aspects of the methodological framework, as every research project can impact on different categories and subgroups. This review provides information that can assist researchers to better demonstrate impact, potentially increasing the likelihood of conducting impactful research and reducing research waste. Existing reviews have not presented a methodological framework that includes different pathways to impact, health impact categories, subgroups, and metrics in a single methodological framework.

Academic-orientated frameworks included in this review advocated the measurement of impact predominantly using so-called ‘quantitative’ metrics—for example, the number of peer-reviewed articles, journal impact factor, and citation rates. This may be because they are well-established measures, relatively easy to capture and objective, and are supported by research funding systems. However, these metrics primarily measure the dissemination of research finding rather than its impact [ 30 , 68 ]. Whilst it is true that wider dissemination, especially when delivered via world-leading international journals, may well lead eventually to changes in healthcare, this is by no means certain. For instance, case studies evaluated by Flinders University of Australia demonstrated that some research projects with non-peer-reviewed publications led to significant changes in health policy, whilst the studies with peer-reviewed publications did not result in any type of impact [ 68 ]. As a result, contemporary literature has tended to advocate the collection of information regarding a variety of different potential forms of impact alongside publication/citations metrics [ 2 , 3 , 5 , 7 , 8 , 29 – 47 ], as outlined in this review.

The 2014 REF exercise adjusted UK university research funding allocation based on evidence of the wider impact of research (through case narrative studies and quantitative metrics), rather than simply according to the quality of research [ 12 ]. The intention was to ensure funds were directed to high-quality research that could demonstrate actual realised benefit. The inclusion of a mixed-method approach to the measurement of impact in the REF (narrative and quantitative metrics) reflects a widespread belief—expressed by the majority of authors of the included methodological frameworks in the review—that individual quantitative impact metrics (e.g., number of citations and publications) do not necessary capture the complexity of the relationships involved in a research project and may exclude measurement of specific aspects of the research pathway [ 10 , 12 ].

Many of the frameworks included in this review advocated the collection of a range of academic, societal, economic, and cultural impact metrics; this is consistent with recent recommendations from the Stern review [ 10 ]. However, a number of these metrics encounter research ‘lag’: i.e., the time between the point at which the research is conducted and when the actual benefits arise [ 69 ]. For instance, some cardiovascular research has taken up to 25 years to generate impact [ 70 ]. Likewise, the impact may not arise exclusively from a single piece of research. Different processes (such as networking interactions and knowledge and research translation) and multiple individuals and organisations are often involved [ 4 , 71 ]. Therefore, attributing the contribution made by each of the different actors involved in the process can be a challenge [ 4 ]. An additional problem associated to attribution is the lack of evidence to link research and impact. The outcomes of research may emerge slowly and be absorbed gradually. Consequently, it is difficult to determine the influence of research in the development of a new policy, practice, or guidelines [ 4 , 23 ].

A further problem is that impact evaluation is conducted ‘ex post’, after the research has concluded. Collecting information retrospectively can be an issue, as the data required might not be available. ‘ex ante’ assessment is vital for funding allocation, as it is necessary to determine the potential forthcoming impact before research is carried out [ 69 ]. Additionally, ex ante evaluation of potential benefit can overcome the issues regarding identifying and capturing evidence, which can be used in the future [ 4 ]. In order to conduct ex ante evaluation of potential benefit, some authors suggest the early involvement of policy makers in a research project coupled with a well-designed strategy of dissemination [ 40 , 69 ].

Providing an alternate view, the authors of methodological frameworks such as the SIAMPI, Contribution Mapping, Research Contribution, and the Exchange model suggest that the problems of attribution are a consequence of assigning the impact of research to a particular impact metric [ 7 , 40 , 42 , 44 ]. To address these issues, these authors propose focusing on the contribution of research through assessing the processes and interactions between stakeholders and researchers, which arguably take into consideration all the processes and actors involved in a research project [ 7 , 40 , 42 , 43 ]. Additionally, contributions highlight the importance of the interactions between stakeholders and researchers from an early stage in the research process, leading to a successful ex ante and ex post evaluation by setting expected impacts and determining how the research outcomes have been utilised, respectively [ 7 , 40 , 42 , 43 ]. However, contribution metrics are generally harder to measure in comparison to academic-orientated indicators [ 72 ].

Currently, there is a debate surrounding the optimal methodological impact framework, and no tool has proven superior to another. The most appropriate methodological framework for a given study will likely depend on stakeholder needs, as each employs different methodologies to assess research impact [ 4 , 37 , 41 ]. This review allows researchers to select individual existing methodological framework components to create a bespoke tool with which to facilitate optimal study design and maximise the potential for impact depending on the characteristic of their study ( Fig 2 and Fig 3 ). For instance, if researchers are interested in assessing how influential their research is on policy making, perhaps considering a suite of the appropriate metrics drawn from multiple methodological frameworks may provide a more comprehensive method than adopting a single methodological framework. In addition, research teams may wish to use a multidimensional approach to methodological framework development, adopting existing narratives and quantitative metrics, as well as elements from contribution frameworks. This approach would arguably present a more comprehensive method of impact assessment; however, further research is warranted to determine its effectiveness [ 4 , 69 , 72 , 73 ].

Finally, it became clear during this review that the included methodological frameworks had been constructed using varied methodological processes. At present, there are no guidelines or consensus around the optimal pathway that should be followed to develop a robust methodological framework. The authors believe this is an area that should be addressed by the research community, to ensure future frameworks are developed using best-practice methodology.

For instance, the Payback Framework drew upon a literature review and was refined through a case study approach. Arguably, this approach could be considered inferior to other methods that involved extensive stakeholder involvement, such as the CIHR framework [ 8 ]. Nonetheless, 7 methodological frameworks were developed based upon the Payback Framework [ 8 , 29 , 31 – 35 ].

Limitations

The present review is the first to summarise systematically existing impact methodological frameworks and metrics. The main limitation is that 50% of the included publications were found through methods other than bibliographic databases searching, indicating poor indexing. Therefore, some relevant articles may not have been included in this review if they failed to indicate the inclusion of a methodological impact framework in their title/abstract. We did, however, make every effort to try to find these potentially hard-to-reach publications, e.g., through forwards/backwards citation searching, hand searching reference lists, and expert communication. Additionally, this review only extracted information regarding the methodology followed to develop each framework from the main publication source or framework webpage. Therefore, further evaluations may not have been included, as they are beyond the scope of the current paper. A further limitation was that although our search strategy did not include language restrictions, we did not specifically search non-English language databases. Thus, we may have failed to identify potentially relevant methodological frameworks that were developed in a non-English language setting.

In conclusion, the measurement of research impact is an essential exercise to help direct the allocation of limited research resources, to maximise benefit, and to help minimise research waste. This review provides a collective summary of existing methodological impact frameworks and metrics, which funders may use to inform the measurement of research impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their research.

Supporting information

S1 appendix. search strategy..

https://doi.org/10.1371/journal.pmed.1002370.s001

S1 PRISMA Checklist. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist.

https://doi.org/10.1371/journal.pmed.1002370.s002

Acknowledgments

We would also like to thank Mrs Susan Bayliss, Information Specialist, University of Birmingham, and Mrs Karen Biddle, Research Secretary, University of Birmingham.

• View Article
• PubMed/NCBI
• Google Scholar
• 3. HEFCE. REF 2014: Assessment framework and guidance on submissions 2011 [cited 2016 15 Feb]. Available from: http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf .
• 8. Canadian Institutes of Health Research. Developing a CIHR framework to measure the impact of health research 2005 [cited 2016 26 Feb]. Available from: http://publications.gc.ca/collections/Collection/MR21-65-2005E.pdf .
• 9. HEFCE. HEFCE allocates £3.97 billion to universities and colleges in England for 2015–1 2015. Available from: http://www.hefce.ac.uk/news/newsarchive/2015/Name,103785,en.html .
• 10. Stern N. Building on Success and Learning from Experience—An Independent Review of the Research Excellence Framework 2016 [cited 2016 05 Aug]. Available from: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/541338/ind-16-9-ref-stern-review.pdf .
• 11. Matthews D. REF sceptic to lead review into research assessment: Times Higher Education; 2015 [cited 2016 21 Apr]. Available from: https://www.timeshighereducation.com/news/ref-sceptic-lead-review-research-assessment .
• 12. HEFCE. The Metric Tide—Report of the Independent Review of the Role of Metrics in Research Assessment and Management 2015 [cited 2016 11 Aug]. Available from: http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/Independentresearch/2015/The,Metric,Tide/2015_metric_tide.pdf .
• 14. LSE Public Policy Group. Maximizing the impacts of your research: A handbook for social scientists. http://www.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.pdf . London: LSE; 2011.
• 15. HEFCE. Consultation on the second Research Excellence Framework. 2016.
• 18. Merriam-Webster Dictionary 2017. Available from: https://www.merriam-webster.com/dictionary/methodology .
• 19. Oxford Dictionaries—pathway 2016 [cited 2016 19 June]. Available from: http://www.oxforddictionaries.com/definition/english/pathway .
• 20. Oxford Dictionaries—metric 2016 [cited 2016 15 Sep]. Available from: https://en.oxforddictionaries.com/definition/metric .
• 21. WHO. WHO Ethical and Safety Guidelines for Interviewing Trafficked Women 2003 [cited 2016 29 July]. Available from: http://www.who.int/mip/2003/other_documents/en/Ethical_Safety-GWH.pdf .
• 31. Kalucy L, et al. Primary Health Care Research Impact Project: Final Report Stage 1 Adelaide: Primary Health Care Research & Information Service; 2007 [cited 2016 26 Feb]. Available from: http://www.phcris.org.au/phplib/filedownload.php?file=/elib/lib/downloaded_files/publications/pdfs/phcris_pub_3338.pdf .
• 33. Canadian Academy of Health Sciences. Making an impact—A preferred framework and indicators to measure returns on investment in health research 2009 [cited 2016 26 Feb]. Available from: http://www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf .
• 39. HEFCE. RAE 2008—Guidance in submissions 2005 [cited 2016 15 Feb]. Available from: http://www.rae.ac.uk/pubs/2005/03/rae0305.pdf .
• 41. Royal Netherlands Academy of Arts and Sciences. The societal impact of applied health research—Towards a quality assessment system 2002 [cited 2016 29 Feb]. Available from: https://www.knaw.nl/en/news/publications/the-societal-impact-of-applied-health-research/@@download/pdf_file/20021098.pdf .
• 48. Weiss CH. Using social research in public policy making: Lexington Books; 1977.
• 50. Kogan M, Henkel M. Government and research: the Rothschild experiment in a government department: Heinemann Educational Books; 1983.
• 51. Thomas P. The Aims and Outcomes of Social Policy Research. Croom Helm; 1985.
• 52. Bulmer M. Social Science Research and Government: Comparative Essays on Britain and the United States: Cambridge University Press; 2010.
• 53. Booth T. Developing Policy Research. Aldershot, Gower1988.
• 55. Kalucy L, et al Exploring the impact of primary health care research Stage 2 Primary Health Care Research Impact Project Adelaide: Primary Health Care Research & Information Service (PHCRIS); 2009 [cited 2016 26 Feb]. Available from: http://www.phcris.org.au/phplib/filedownload.php?file=/elib/lib/downloaded_files/publications/pdfs/phcris_pub_8108.pdf .
• 56. CHSRF. Canadian Health Services Research Foundation 2000. Health Services Research and Evidence-based Decision Making [cited 2016 February]. Available from: http://www.cfhi-fcass.ca/migrated/pdf/mythbusters/EBDM_e.pdf .
• 58. W.K. Kellogg Foundation. Logic Model Development Guide 2004 [cited 2016 19 July]. Available from: http://www.smartgivers.org/uploads/logicmodelguidepdf.pdf .
• 59. United Way of America. Measuring Program Outcomes: A Practical Approach 1996 [cited 2016 19 July]. Available from: https://www.bttop.org/sites/default/files/public/W.K.%20Kellogg%20LogicModel.pdf .
• 60. Nutley S, Percy-Smith J and Solesbury W. Models of research impact: a cross sector review of literature and practice. London: Learning and Skills Research Centre 2003.
• 61. Spaapen J, van Drooge L. SIAMPI final report [cited 2017 Jan]. Available from: http://www.siampi.eu/Content/SIAMPI_Final%20report.pdf .
• 63. LSHTM. The Health Risks and Consequences of Trafficking in Women and Adolescents—Findings from a European Study 2003 [cited 2016 29 July]. Available from: http://www.oas.org/atip/global%20reports/zimmerman%20tip%20health.pdf .
• 70. Russell G. Response to second HEFCE consultation on the Research Excellence Framework 2009 [cited 2016 04 Apr]. Available from: http://russellgroup.ac.uk/media/5262/ref-consultation-response-final-dec09.pdf .

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Perspective
• Open access
• Published: 09 April 2024

The potential for artificial intelligence to transform healthcare: perspectives from international health leaders

• Christina Silcox 1 ,
• Eyal Zimlichmann 2 , 3 ,
• Katie Huber   ORCID: orcid.org/0000-0003-2519-8714 1 ,
• Neil Rowen 1 ,
• Robert Saunders 1 ,
• Mark McClellan 1 ,
• Charles N. Kahn III 3 , 4 ,
• Claudia A. Salzberg 3 &
• David W. Bates   ORCID: orcid.org/0000-0001-6268-1540 5 , 6 , 7  

npj Digital Medicine volume  7 , Article number:  88 ( 2024 ) Cite this article

9459 Accesses

6 Citations

43 Altmetric

Metrics details

• Health policy
• Health services

Artificial intelligence (AI) has the potential to transform care delivery by improving health outcomes, patient safety, and the affordability and accessibility of high-quality care. AI will be critical to building an infrastructure capable of caring for an increasingly aging population, utilizing an ever-increasing knowledge of disease and options for precision treatments, and combatting workforce shortages and burnout of medical professionals. However, we are not currently on track to create this future. This is in part because the health data needed to train, test, use, and surveil these tools are generally neither standardized nor accessible. There is also universal concern about the ability to monitor health AI tools for changes in performance as they are implemented in new places, used with diverse populations, and over time as health data may change. The Future of Health (FOH), an international community of senior health care leaders, collaborated with the Duke-Margolis Institute for Health Policy to conduct a literature review, expert convening, and consensus-building exercise around this topic. This commentary summarizes the four priority action areas and recommendations for health care organizations and policymakers across the globe that FOH members identified as important for fully realizing AI’s potential in health care: improving data quality to power AI, building infrastructure to encourage efficient and trustworthy development and evaluations, sharing data for better AI, and providing incentives to accelerate the progress and impact of AI.

Similar content being viewed by others

Guiding principles for the responsible development of artificial intelligence tools for healthcare

A short guide for medical professionals in the era of artificial intelligence

Innovation and challenges of artificial intelligence technology in personalized healthcare, introduction.

Artificial intelligence (AI), supported by timely and accurate data and evidence, has the potential to transform health care delivery by improving health outcomes, patient safety, and the affordability and accessibility of high-quality care 1 , 2 . AI integration is critical to building an infrastructure capable of caring for an increasingly aging population, utilizing an ever-increasing knowledge of disease and options for precision treatments, and combatting workforce shortages and burnout of medical professionals. However, we are not currently on track to create this future. This is in part because the health data needed to train, test, use, and surveil these tools are generally neither standardized nor accessible. This is true across the international community, although there is variable progress within individual countries. There is also universal concern about monitoring health AI tools for changes in performance as they are implemented in new places, used with diverse populations, and over time as health data may change.

The Future of Health (FOH) is an international community of senior health care leaders representing health systems, health policy, health care technology, venture funding, insurance, and risk management. FOH collaborated with the Duke-Margolis Institute for Health Policy to conduct a literature review, expert convening, and consensus-building exercise. In total, 46 senior health care leaders were engaged in this work, from eleven countries in Europe, North America, Africa, Asia, and Australia. This commentary summarizes the four priority action areas and recommendations for health care organizations and policymakers that FOH members identified as important for fully realizing AI’s potential in health care: improving data quality to power AI, building infrastructure to encourage efficient and trustworthy development and evaluations, sharing data for better AI, and providing incentives to accelerate the progress and impact of AI.

Powering AI through high-quality data

“Going forward, data are going to be the most valuable commodity in health care. Organizations need robust plans about how to mobilize and use their data.”

AI algorithms will only perform as well as the accuracy and completeness of key underlying data, and data quality is dependent on actions and workflows that encourage trust.

To begin to improve data quality, FOH members agreed that an initial priority is identifying and assuring reliable availability of high-priority data elements for promising AI applications: those with the most predictive value, those of the highest value to patients, and those most important for analyses of performance, including subgroup analyses to detect bias.

Leaders should also advocate for aligned policy incentives to improve the availability and reliability of these priority data elements. There are several examples of efforts across the world to identify and standardize high-priority data elements for AI applications and beyond, such as the multinational project STANDING Together, which is developing standards to improve the quality and representativeness of data used to build and test AI tools 3 .

Policy incentives that would further encourage high-quality data collection include (1) aligned payment incentives for measures of health care quality and safety, and ensuring the reliability of the underlying data, and (2) quality measures and performance standards focused on the reliability, completeness, and timeliness of collection and sharing of high-priority data itself.

Trust and verify

“Your AI algorithms are only going to be as good as the data and the real-world evidence used to validate them, and the data are only going to be as good as the trust and privacy and supporting policies.”

FOH members stressed the importance of showing that AI tools are both effective and safe within their specific patient populations.

This is a particular challenge with AI tools, whose performance can differ dramatically across sites and over time, as health data patterns and population characteristics vary. For example, several studies of the Epic Sepsis Model found both location-based differences in performance and degradation in performance over time due to data drift 4 , 5 . However, real-world evaluations are often much more difficult for algorithms that are used for longer-term predictions, or to avert long-term complications from occurring, particularly in the absence of connected, longitudinal data infrastructure. As such, health systems must prioritize implementing data standards and data infrastructure that can facilitate the retraining or tuning of algorithms, test for local performance and bias, and ensure scalability across the organization and longer-term applications 6 .

There are efforts to help leaders and health systems develop consensus-based evaluation techniques and infrastructure for AI tools, including HealthAI: The Global Agency for Responsible AI in Health, which aims to build and certify validation mechanisms for nations and regions to adopt; and the Coalition for Health AI (CHAI), which recently announced plans to build a US-wide health AI assurance labs network 7 , 8 . These efforts, if successful, will assist manufacturers and health systems in complying with new laws, rules, and regulations being proposed and released that seek to ensure AI tools are trustworthy, such as the EU AI Act and the 2023 US Executive Order on AI.

Sharing data for better AI

“Underlying these challenges is the investment required to standardize business processes so that you actually get data that’s usable between institutions and even within an institution.”

While high-quality internal data may enable some types of AI-tool development and testing, this is insufficient to power and evaluate all AI applications. To build truly effective AI-enabled predictive software for clinical care and predictive supports, data often need to be interoperable across health systems to build a diverse picture of patients’ health across geographies, and reliably shared.

FOH members recommended that health care leaders work with researchers and policymakers to connect detailed encounter data with longitudinal outcomes, and pilot opportunities across diverse populations and systems to help assure valid outcome evaluations as well as address potential confounding and population subgroup differences—the ability to aggregate data is a clear rate-limiting step. The South African National Digital Health Strategy outlined interventions to improve the adoption of digital technologies while complying with the 2013 Protection of Personal Information Act 9 . Although challenges remain, the country has made progress on multiple fronts, including building out a Health Patient Registration System as a first step towards a portable, longitudinal patient record system and releasing a Health Normative Standards Framework to improve data flow across institutional and geographic boundaries 10 .

Leaders should adopt policies in their organizations, and encourage adoption in their province and country, that simplify data governance and sharing while providing appropriate privacy protections – including building foundations of trust with patients and the public as previously discussed. Privacy-preserving innovations include ways to “share” data without movement from protected systems using approaches like federated analyses, data sandboxes, or synthetic data. In addition to exploring privacy-preserving approaches to data sharing, countries and health systems may need to consider broad and dynamic approaches to consent 11 , 12 . As we look to a future where a patient may have thousands of algorithms churning away at their data, efforts to improve data quality and sharing should include enabling patients’ access to and engagement with their own data to encourage them to actively partner in their health and provide transparency on how their data are being used to improve health care. For example, the Understanding Patient Data program in the United Kingdom produces research and resources to explain how the National Health Service uses patients’ data 13 . Community engagement efforts can further assist with these efforts by building trust and expanding understanding.

FOH members also stressed the importance of timely data access. Health systems should work together to establish re-usable governance and privacy frameworks that allow stakeholders to clearly understand what data will be shared and how it will be protected to reduce the time needed for data use agreements. Trusted third-party data coordinating centers could also be used to set up “precertification” systems around data quality, testing, and cybersecurity to support health organizations with appropriate data stewardship to form partnerships and access data rapidly.

Incentivizing progress for AI impact

“Unless it’s tied to some kind of compensation to the organization, the drive to help implement those tools and overcome that risk aversion is going to be very high… I do think that business driver needs to be there.”

AI tools and data quality initiatives have not moved as quickly in health care due to the lack of direct payment, and often, misalignment of financial incentives and supports for high-quality data collection and predictive analytics. This affects both the ability to purchase and safely implement commercial AI products as well as the development of “homegrown” AI tools.

FOH members recommended that leaders should advocate for paying for value in health – quality, safety, better health, and lower costs for patients. This better aligns the financial incentives for accelerating the development, evaluation, and adoption of AI as well as other tools designed to either keep patients healthy or quickly diagnose and treat them with the most effective therapies when they do become ill. Effective personalized health care requires high-quality, standardized, interoperable datasets from diverse sources 14 . Within value-based payments themselves, data are critical to measuring quality of care and patient outcomes, adjusted or contextualized for factors outside of clinical control. Value-based payments therefore align incentives for (1) high-quality data collection and trusted use, (2) building effective AI tools, and (3) ensuring that those tools are improving patient outcomes and/or health system operations.

Data have become the most valuable commodity in health care, but questions remain about whether there will be an AI “revolution” or “evolution” in health care delivery. Early AI applications in certain clinical areas have been promising, but more advanced AI tools will require higher quality, real-world data that is interoperable and secure. The steps health care organization leaders and policymakers take in the coming years, starting with short-term opportunities to develop meaningful AI applications that achieve measurable improvements in outcomes and costs, will be critical in enabling this future that can improve health outcomes, safety, affordability, and equity.

Data availability

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Abernethy, A. et al. The promise of digital health: then, now, and the future. NAM Perspect. 6 (2022).

Akpakwu, E. Four ways AI can make healthcare more efficient and affordable. World Economic Forum https://www.weforum.org/agenda/2018/05/four-ways-ai-is-bringing-down-the-cost-of-healthcare/ (2018).

STANDING Together. https://www.datadiversity.org/home .

Wong, A. et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med 181 , 1065–1070 (2021).

Article   PubMed   Google Scholar  

Ross, C. STAT and MIT rooted out the weaknesses in health care algorithms. Here’s how we did it. STAT https://www.statnews.com/2022/02/28/data-drift-machine-learning/ (2022).

Locke, T., Parker, V., Thoumi, A., Goldstein, B. & Silcox, C. Preventing bias and inequities in AI-enabled health tools . https://healthpolicy.duke.edu/publications/preventing-bias-and-inequities-ai-enabled-health-tools (2022).

Introducing HealthAI. The International Digital Health and AI Research Collaborative (I-DAIR) https://www.i-dair.org/news/introducing-healthai (2023).

Shah, N. H. et al. A nationwide network of health AI assurance laboratories. JAMA 331 , 245 (2024).

Singh, V. AI & Data in South Africa’s Health Sector . https://policyaction.org.za/sites/default/files/PAN_TopicalGuide_AIData6_Health_Elec.pdf (2020).

Zharima, C., Griffiths, F. & Goudge, J. Exploring the barriers and facilitators to implementing electronic health records in a middle-income country: a qualitative study from South Africa. Front. Digit. Health 5 , 1207602 (2023).

Article   PubMed   PubMed Central   Google Scholar  

Lee, A. R. et al. Identifying facilitators of and barriers to the adoption of dynamic consent in digital health ecosystems: a scoping review. BMC Med. Ethics 24 , 107 (2023).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Stoeklé, H. C., Hulier-Ammar, E. & Hervé, C. Data medicine: ‘broad’ or ‘dynamic’ consent? Public Health Ethics 15 , 181–185 (2022).

Article   Google Scholar  

Understanding Patient Data. Understanding Patient Data http://understandingpatientdata.org.uk/ .

Chén, O. Y. & Roberts, B. Personalized health care and public health in the digital age. Front. Digit. Health 3 , 595704 (2021).

Download references

Acknowledgements

The authors acknowledge Oranit Ido and Jonathan Gonzalez-Smith for their contributions to this work. This study was funded by The Future of Health, LLC. The Future of Health, LLC, was involved in all stages of this research, including study design, data collection, analysis and interpretation of data, and the preparation of this manuscript.

Author information

Authors and affiliations.

Duke-Margolis Institute for Health Policy, Duke University, Washington, DC, USA &, Durham, NC, USA

Christina Silcox, Katie Huber, Neil Rowen, Robert Saunders & Mark McClellan

Sheba Medical Center, Ramat Gan, Israel

Eyal Zimlichmann

Future of Health, Washington, DC, USA

Eyal Zimlichmann, Charles N. Kahn III & Claudia A. Salzberg

Federation of American Hospitals, Washington, DC, USA

Charles N. Kahn III

Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, MA, USA

David W. Bates

Harvard Medical School, Boston, MA, USA

Department of Health Policy and Management, Harvard T. H. Chan School of Public Health, Boston, MA, USA

You can also search for this author in PubMed   Google Scholar

Contributions

C.S., K.H., N.R., and R.S. conducted initial background research and analyzed qualitative data from stakeholders. All authors (C.S., E.Z., K.H., N.R., R.S., M.M., C.K., C.A.S., and D.B.) assisted with conceptualization of the project and strategic guidance. C.S., K.H., and N.R. wrote initial drafts of the manuscript. All authors contributed to critical revisions of the manuscript and read and approved the final manuscript.

Corresponding author

Correspondence to David W. Bates .

Ethics declarations

Competing interests.

C.S., K.H., N.R., and C.A.S. declare no competing interests. E.Z. reports personal fees from Arkin Holdings, personal fees from Statista and equity from Valera Health, Profility and Hello Heart. R.S. has been an external reviewer for The John A. Hartford Foundation, and is a co-chair for the Health Evolution Summit Roundtable on Value-Based Care for Specialized Populations. M.M. is an independent director on the boards of Johnson & Johnson, Cigna, Alignment Healthcare, and PrognomIQ; co-chairs the Guiding Committee for the Health Care Payment Learning and Action Network; and reports fees for serving as an adviser for Arsenal Capital Partners, Blackstone Life Sciences, and MITRE. C.K. is a Profility Board member and additionally reports equity from Valera Health and MDClone. D.W.B. reports grants and personal fees from EarlySense, personal fees from CDI Negev, equity from Valera Health, equity from Clew, equity from MDClone, personal fees and equity from AESOP, personal fees and equity from Feelbetter, equity from Guided Clinical Solutions, and grants from IBM Watson Health, outside the submitted work. D.W.B. has a patent pending (PHC-028564 US PCT), on intraoperative clinical decision support.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Silcox, C., Zimlichmann, E., Huber, K. et al. The potential for artificial intelligence to transform healthcare: perspectives from international health leaders. npj Digit. Med. 7 , 88 (2024). https://doi.org/10.1038/s41746-024-01097-6

Download citation

Received : 30 October 2023

Accepted : 29 March 2024

Published : 09 April 2024

DOI : https://doi.org/10.1038/s41746-024-01097-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Predicting childhood outcomes through standardizing disease definitions and promoting multicenter enrollment.

• Fu-Sheng Chou
• Ashwini Lakshmanan

Pediatric Research (2024)

Visual–language foundation models in medicine

The Visual Computer (2024)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Research article
• Open access
• Published: 10 April 2021

The role of artificial intelligence in healthcare: a structured literature review

• Silvana Secinaro 1 ,
• Davide Calandra 1 ,
• Aurelio Secinaro 2 ,
• Vivek Muthurangu 3 &
• Paolo Biancone 1  

BMC Medical Informatics and Decision Making volume  21 , Article number:  125 ( 2021 ) Cite this article

164k Accesses

313 Citations

28 Altmetric

Metrics details

Background/Introduction

Artificial intelligence (AI) in the healthcare sector is receiving attention from researchers and health professionals. Few previous studies have investigated this topic from a multi-disciplinary perspective, including accounting, business and management, decision sciences and health professions.

The structured literature review with its reliable and replicable research protocol allowed the researchers to extract 288 peer-reviewed papers from Scopus. The authors used qualitative and quantitative variables to analyse authors, journals, keywords, and collaboration networks among researchers. Additionally, the paper benefited from the Bibliometrix R software package.

The investigation showed that the literature in this field is emerging. It focuses on health services management, predictive medicine, patient data and diagnostics, and clinical decision-making. The United States, China, and the United Kingdom contributed the highest number of studies. Keyword analysis revealed that AI can support physicians in making a diagnosis, predicting the spread of diseases and customising treatment paths.

Conclusions

The literature reveals several AI applications for health services and a stream of research that has not fully been covered. For instance, AI projects require skills and data quality awareness for data-intensive analysis and knowledge-based management. Insights can help researchers and health professionals understand and address future research on AI in the healthcare field.

Peer Review reports

Artificial intelligence (AI) generally applies to computational technologies that emulate mechanisms assisted by human intelligence, such as thought, deep learning, adaptation, engagement, and sensory understanding [ 1 , 2 ]. Some devices can execute a role that typically involves human interpretation and decision-making [ 3 , 4 ]. These techniques have an interdisciplinary approach and can be applied to different fields, such as medicine and health. AI has been involved in medicine since as early as the 1950s, when physicians made the first attempts to improve their diagnoses using computer-aided programs [ 5 , 6 ]. Interest and advances in medical AI applications have surged in recent years due to the substantially enhanced computing power of modern computers and the vast amount of digital data available for collection and utilisation [ 7 ]. AI is gradually changing medical practice. There are several AI applications in medicine that can be used in a variety of medical fields, such as clinical, diagnostic, rehabilitative, surgical, and predictive practices. Another critical area of medicine where AI is making an impact is clinical decision-making and disease diagnosis. AI technologies can ingest, analyse, and report large volumes of data across different modalities to detect disease and guide clinical decisions [ 3 , 8 ]. AI applications can deal with the vast amount of data produced in medicine and find new information that would otherwise remain hidden in the mass of medical big data [ 9 , 10 , 11 ]. These technologies can also identify new drugs for health services management and patient care treatments [ 5 , 6 ].

Courage in the application of AI is visible through a search in the primary research databases. However, as Meskò et al. [ 7 ] find, the technology will potentially reduce care costs and repetitive operations by focusing the medical profession on critical thinking and clinical creativity. As Cho et al. and Doyle et al. [ 8 , 9 ] add, the AI perspective is exciting; however, new studies will be needed to establish the efficacy and applications of AI in the medical field [ 10 ].

Our paper will also concentrate on AI strategies for healthcare from the accounting, business, and management perspectives. The authors used the structured literature review (SLR) method for its reliable and replicable research protocol [ 11 ] and selected bibliometric variables as sources of investigation. Bibliometric usage enables the recognition of the main quantitative variables of the study stream [ 12 ]. This method facilitates the detection of the required details of a particular research subject, including field authors, number of publications, keywords for interaction between variables (policies, properties and governance) and country data [ 13 ]. It also allows the application of the science mapping technique [ 14 ]. Our paper adopted the Bibliometrix R package and the biblioshiny web interface as tools of analysis [ 14 ].

The investigation offers the following insights for future researchers and practitioners:

bibliometric information on 288 peer-reviewed English papers from the Scopus collection.

Identification of leading journals in this field, such as Journal of Medical Systems, Studies in Health Technology and Informatics, IEEE Journal of Biomedical and Health Informatics, and Decision Support Systems.

Qualitative and quantitative information on authors’ Lotka’s law, h-index, g-index, m-index, keyword, and citation data.

Research on specific countries to assess AI in the delivery and effectiveness of healthcare, quotes, and networks within each region.

A topic dendrogram study that identifies five research clusters: health services management, predictive medicine, patient data, diagnostics, and finally, clinical decision-making.

An in-depth discussion that develops theoretical and practical implications for future studies.

The paper is organised as follows. Section  2 lists the main bibliometric articles in this field. Section  3 elaborates on the methodology. Section  4 presents the findings of the bibliometric analysis. Section  5 discusses the main elements of AI in healthcare based on the study results. Section  6 concludes the article with future implications for research.

Related works and originality

As suggested by Zupic and Čater [ 15 ], a research stream can be evaluated with bibliometric methods that can introduce objectivity and mitigate researcher bias. For this reason, bibliometric methods are attracting increasing interest among researchers as a reliable and impersonal research analytical approach [ 16 , 17 ]. Recently, bibliometrics has been an essential method for analysing and predicting research trends [ 18 ]. Table  1 lists other research that has used a similar approach in the research stream investigated.

The scientific articles reported show substantial differences in keywords and research topics that have been previously studied. The bibliometric analysis of Huang et al. [ 19 ] describes rehabilitative medicine using virtual reality technology. According to the authors, the primary goal of rehabilitation is to enhance and restore functional ability and quality of life for patients with physical impairments or disabilities. In recent years, many healthcare disciplines have been privileged to access various technologies that provide tools for both research and clinical intervention.

Hao et al. [ 20 ] focus on text mining in medical research. As reported, text mining reveals new, previously unknown information by using a computer to automatically extract information from different text resources. Text mining methods can be regarded as an extension of data mining to text data. Text mining is playing an increasingly significant role in processing medical information. Similarly, the studies by dos Santos et al. [ 21 ] focus on applying data mining and machine learning (ML) techniques to public health problems. As stated in this research, public health may be defined as the art and science of preventing diseases, promoting health, and prolonging life. Using data mining and ML techniques, it is possible to discover new information that otherwise would be hidden. These two studies are related to another topic: medical big data. According to Liao et al. [ 22 ], big data is a typical “buzzword” in the business and research community, referring to a great mass of digital data collected from various sources. In the medical field, we can obtain a vast amount of data (i.e., medical big data). Data mining and ML techniques can help deal with this information and provide helpful insights for physicians and patients. More recently, Choudhury et al. [ 23 ] provide a systematic review on the use of ML to improve the care of elderly patients, demonstrating eligible studies primarily in psychological disorders and eye diseases.

Tran et al. [ 2 ] focus on the global evolution of AI research in medicine. Their bibliometric analysis highlights trends and topics related to AI applications and techniques. As stated in Connelly et al.’s [ 24 ] study, robot-assisted surgeries have rapidly increased in recent years. Their bibliometric analysis demonstrates how robotic-assisted surgery has gained acceptance in different medical fields, such as urological, colorectal, cardiothoracic, orthopaedic, maxillofacial and neurosurgery applications. Additionally, the bibliometric analysis of Guo et al. [ 25 ] provides an in-depth study of AI publications through December 2019. The paper focuses on tangible AI health applications, giving researchers an idea of how algorithms can help doctors and nurses. A new stream of research related to AI is also emerging. In this sense, Choudhury and Asan’s [ 26 ] scientific contribution provides a systematic review of the AI literature to identify health risks for patients. They report on 53 studies involving technology for clinical alerts, clinical reports, and drug safety. Considering the considerable interest within this research stream, this analysis differs from the current literature for several reasons. It aims to provide in-depth discussion, considering mainly the business, management, and accounting fields and not dealing only with medical and health profession publications.

Additionally, our analysis aims to provide a bibliometric analysis of variables such as authors, countries, citations and keywords to guide future research perspectives for researchers and practitioners, as similar analyses have done for several publications in other research streams [ 15 , 16 , 27 ]. In doing so, we use a different database, Scopus, that is typically adopted in social sciences fields. Finally, our analysis will propose and discuss a dominant framework of variables in this field, and our analysis will not be limited to AI application descriptions.

Methodology

This paper evaluated AI in healthcare research streams using the SLR method [ 11 ]. As suggested by Massaro et al. [ 11 ], an SLR enables the study of the scientific corpus of a research field, including the scientific rigour, reliability and replicability of operations carried out by researchers. As suggested by many scholars, the methodology allows qualitative and quantitative variables to highlight the best authors, journals and keywords and combine a systematic literature review and bibliometric analysis [ 27 , 28 , 29 , 30 ]. Despite its widespread use in business and management [ 16 , 31 ], the SLR is also used in the health sector based on the same philosophy through which it was originally conceived [ 32 , 33 ]. A methodological analysis of previously published articles reveals that the most frequently used steps are as follows [ 28 , 31 , 34 ]:

defining research questions;

writing the research protocol;

defining the research sample to be analysed;

developing codes for analysis; and

critically analysing, discussing, and identifying a future research agenda.

Considering the above premises, the authors believe that an SLR is the best method because it combines scientific validity, replicability of the research protocol and connection between multiple inputs.

As stated by the methodological paper, the first step is research question identification. For this purpose, we benefit from the analysis of Zupic and Čater [ 15 ], who provide several research questions for future researchers to link the study of authors, journals, keywords and citations. Therefore, RQ1 is “What are the most prominent authors, journal keywords and citations in the field of the research study?” Additionally, as suggested by Haleem et al. [ 35 ], new technologies, including AI, are changing the medical field in unexpected timeframes, requiring studies in multiple areas. Therefore, RQ2 is “How does artificial intelligence relate to healthcare, and what is the focus of the literature?” Then, as discussed by Massaro et al. [ 36 ], RQ3 is “What are the research applications of artificial intelligence for healthcare?”.

The first research question aims to define the qualitative and quantitative variables of the knowledge flow under investigation. The second research question seeks to determine the state of the art and applications of AI in healthcare. Finally, the third research question aims to help researchers identify practical and theoretical implications and future research ideas in this field.

The second fundamental step of the SLR is writing the research protocol [ 11 ]. Table  2 indicates the currently known literature elements, uniquely identifying the research focus, motivations and research strategy adopted and the results providing a link with the following points. Additionally, to strengthen the analysis, our investigation benefits from the PRISMA statement methodological article [ 37 ]. Although the SLR is a validated method for systematic reviews and meta-analyses, we believe that the workflow provided may benefit the replicability of the results [ 37 , 38 , 39 , 40 ]. Figure  1 summarises the researchers’ research steps, indicating that there are no results that can be referred to as a meta-analysis.

Source : Authors’ elaboration on Liberati et al. [ 37 ]

PRISMA workflow.

The third step is to specify the search strategy and search database. Our analysis is based on the search string “Artificial Intelligence” OR “AI” AND “Healthcare” with a focus on “Business, Management, and Accounting”, “Decision Sciences”, and “Health professions”. As suggested by [ 11 , 41 ] and motivated by [ 42 ], keywords can be selected through a top-down approach by identifying a large search field and then focusing on particular sub-topics. The paper uses data retrieved from the Scopus database, a multi-disciplinary database, which allowed the researchers to identify critical articles for scientific analysis [ 43 ]. Additionally, Scopus was selected based on Guo et al.’s [ 25 ] limitations, which suggest that “future studies will apply other databases, such as Scopus, to explore more potential papers” . The research focuses on articles and reviews published in peer-reviewed journals for their scientific relevance [ 11 , 16 , 17 , 29 ] and does not include the grey literature, conference proceedings or books/book chapters. Articles written in any language other than English were excluded [ 2 ]. For transparency and replicability, the analysis was conducted on 11 January 2021. Using this research strategy, the authors retrieved 288 articles. To strengthen the study's reliability, we publicly provide the full bibliometric extract on the Zenodo repository [ 44 , 45 ].

The fourth research phase is defining the code framework that initiates the analysis of the variables. The study will identify the following:

descriptive information of the research area;

source analysis [ 16 ];

author and citation analysis [ 28 ];

keywords and network analysis [ 14 ]; and

geographic distribution of the papers [ 14 ].

The final research phase is the article’s discussion and conclusion, where implications and future research trends will be identified.

At the research team level, the information is analysed with the statistical software R-Studio and the Bibliometrix package [ 15 ], which allows scientific analysis of the results obtained through the multi-disciplinary database.

The analysis of bibliometric results starts with a description of the main bibliometric statistics with the aim of answering RQ1, What are the most prominent authors, journal keywords and citations in the field of the research study?, and RQ2, How does artificial intelligence relate to healthcare, and what is the focus of the literature? Therefore, the following elements were thoroughly analysed: (1) type of document; (2) annual scientific production; (3) scientific sources; (4) source growth; (5) number of articles per author; (6) author’s dominance ranking; (7) author’s h-index, g-index, and m-index; (8) author’s productivity; (9) author’s keywords; (10) topic dendrogram; (11) a factorial map of the document with the highest contributions; (12) article citations; (13) country production; (14) country citations; (15) country collaboration map; and (16) country collaboration network.

Main information

Table  3 shows the information on 288 peer-reviewed articles published between 1992 and January 2021 extracted from the Scopus database. The number of keywords is 946 from 136 sources, and the number of keywords plus, referring to the number of keywords that frequently appear in an article’s title, was 2329. The analysis period covered 28 years and 1 month of scientific production and included an annual growth rate of 5.12%. However, the most significant increase in published articles occurred in the past three years (please see Fig.  2 ). On average, each article was written by three authors (3.56). Finally, the collaboration index (CI), which was calculated as the total number of authors of multi-authored articles/total number of multi-authored articles, was 3.97 [ 46 ].

Source : Authors’ elaboration

Annual scientific production.

Table  4 shows the top 20 sources related to the topic. The Journal of Medical Systems is the most relevant source, with twenty-one of the published articles. This journal's main issues are the foundations, functionality, interfaces, implementation, impacts, and evaluation of medical technologies. Another relevant source is Studies in Health Technology and Informatics, with eleven articles. This journal aims to extend scientific knowledge related to biomedical technologies and medical informatics research. Both journals deal with cloud computing, machine learning, and AI as a disruptive healthcare paradigm based on recent publications. The IEEE Journal of Biomedical and Health Informatics investigates technologies in health care, life sciences, and biomedicine applications from a broad perspective. The next journal, Decision Support Systems, aims to analyse how these technologies support decision-making from a multi-disciplinary view, considering business and management. Therefore, the analysis of the journals revealed that we are dealing with an interdisciplinary research field. This conclusion is confirmed, for example, by the presence of purely medical journals, journals dedicated to the technological growth of healthcare, and journals with a long-term perspective such as futures.

The distribution frequency of the articles (Fig.  3 ) indicates the journals dealing with the topic and related issues. Between 2008 and 2012, a significant growth in the number of publications on the subject is noticeable. However, the graph shows the results of the Loess regression, which includes the quantity and publication time of the journal under analysis as variables. This method allows the function to assume an unlimited distribution; that is, feature can consider values below zero if the data are close to zero. It contributes to a better visual result and highlights the discontinuity in the publication periods [ 47 ].

Source growth. Source : Authors’ elaboration

Finally, Fig.  4 provides an analytical perspective on factor analysis for the most cited papers. As indicated in the literature [ 48 , 49 ], using factor analysis to discover the most cited papers allows for a better understanding of the scientific world’s intellectual structure. For example, our research makes it possible to consider certain publications that effectively analyse subject specialisation. For instance, Santosh’s [ 50 ] article addresses the new paradigm of AI with ML algorithms for data analysis and decision support in the COVID-19 period, setting a benchmark in terms of citations by researchers. Moving on to the application, an article by Shickel et al. [ 51 ] begins with the belief that the healthcare world currently has much health and administrative data. In this context, AI and deep learning will support medical and administrative staff in extracting data, predicting outcomes, and learning medical representations. Finally, in the same line of research, Baig et al. [ 52 ], with a focus on wearable patient monitoring systems (WPMs), conclude that AI and deep learning may be landmarks for continuous patient monitoring and support for healthcare delivery.

Factorial map of the most cited documents.

This section identifies the most cited authors of articles on AI in healthcare. It also identifies the authors’ keywords, dominance factor (DF) ranking, h-index, productivity, and total number of citations. Table  5 identifies the authors and their publications in the top 20 rankings. As the table shows, Bushko R.G. has the highest number of publications: four papers. He is the editor-in-chief of Future of Health Technology, a scientific journal that aims to develop a clear vision of the future of health technology. Then, several authors each wrote three papers. For instance, Liu C. is a researcher active in the topic of ML and computer vision, and Sharma A. from Emory University Atlanta in the USA is a researcher with a clear focus on imaging and translational informatics. Some other authors have two publications each. While some authors have published as primary authors, most have published as co-authors. Hence, in the next section, we measure the contributory power of each author by investigating the DF ranking through the number of elements.

Authors’ dominance ranking

The dominance factor (DF) is a ratio measuring the fraction of multi-authored articles in which an author acts as the first author [ 53 ]. Several bibliometric studies use the DF in their analyses [ 46 , 54 ]. The DF ranking calculates an author’s dominance in producing articles. The DF is calculated by dividing the number of an author’s multi-authored papers as the first author (Nmf) by the author's total number of multi-authored papers (Nmt). This is omitted in the single-author case due to the constant value of 1 for single-authored articles. This formulation could lead to some distortions in the results, especially in fields where the first author is entered by surname alphabetical order [ 55 ].

The mathematical equation for the DF is shown as:

Table  6 lists the top 20 DF rankings. The data in the table show a low level of articles per author, either for first-authored or multi-authored articles. The results demonstrate that we are dealing with an emerging topic in the literature. Additionally, as shown in the table, Fox J. and Longoni C. are the most dominant authors in the field.

Authors’ impact

Table  7 shows the impact of authors in terms of the h-index [ 56 ] (i.e., the productivity and impact of citations of a researcher), g-index [ 57 ] (i.e., the distribution of citations received by a researcher's publications), m-index [ 58 ] (i.e., the h-index value per year), total citations, total paper and years of scientific publication. The H-index was introduced in the literature as a metric for the objective comparison of scientific results and depended on the number of publications and their impact [ 59 ]. The results show that the 20 most relevant authors have an h-index between 2 and 1. For the practical interpretation of the data, the authors considered data published by the London School of Economics [ 60 ]. In the social sciences, the analysis shows values of 7.6 for economic publications by professors and researchers who had been active for several years. Therefore, the youthfulness of the research area has attracted young researchers and professors. At the same time, new indicators have emerged over the years to diversify the logic of the h-index. For example, the g-index indicates an author's impact on citations, considering that a single article can generate these. The m-index, on the other hand, shows the cumulative value over the years.

The analysis, also considering the total number of citations, the number of papers published and the year of starting to publish, thus confirms that we are facing an expanding research flow.

Authors’ productivity

Figure  5 shows Lotka’s law. This mathematical formulation originated in 1926 to describe the publication frequency by authors in a specific research field [ 61 ]. In practice, the law states that the number of authors contributing to research in a given period is a fraction of the number who make up a single contribution [ 14 , 61 ].

Lotka’s law.

The mathematical relationship is expressed in reverse in the following way:

where y x is equal to the number of authors producing x articles in each research field. Therefore, C and n are constants that can be estimated in the calculation.

The figure's results are in line with Lotka's results, with an average of two publications per author in a given research field. In addition, the figure shows the percentage of authors. Our results lead us to state that we are dealing with a young and growing research field, even with this analysis. Approximately 70% of the authors had published only their first research article. Only approximately 20% had published two scientific papers.

Authors’ keywords

This section provides information on the relationship between the keywords artificial intelligence and healthcare . This analysis is essential to determine the research trend, identify gaps in the discussion on AI in healthcare, and identify the fields that can be interesting as research areas [ 42 , 62 ].

Table  8 highlights the total number of keywords per author in the top 20 positions. The ranking is based on the following elements: healthcare, artificial intelligence, and clinical decision support system . Keyword analysis confirms the scientific area of reference. In particular, we deduce the definition as “Artificial intelligence is the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages” [ 2 , 63 ]. Panch et al. [ 4 ] find that these technologies can be used in different business and management areas. After the first keyword, the analysis reveals AI applications and related research such as machine learning and deep learning.

Additionally, data mining and big data are a step forward in implementing exciting AI applications. According to our specific interest, if we applied AI in healthcare, we would achieve technological applications to help and support doctors and medical researchers in decision-making. The link between AI and decision-making is the reason why we find, in the seventh position, the keyword clinical decision support system . AI techniques can unlock clinically relevant information hidden in the massive amount of data that can assist clinical decision-making [ 64 ]. If we analyse the following keywords, we find other elements related to decision-making and support systems.

The TreeMap below (Fig.  6 ) highlights the combination of possible keywords representing AI and healthcare.

Keywords treemap.

The topic dendrogram in Fig.  7 represents the hierarchical order and the relationship between the keywords generated by hierarchical clustering [ 42 ]. The cut in the figure and the vertical lines facilitate an investigation and interpretation of the different clusters. As stated by Andrews [ 48 ], the figure is not intended to find the perfect level of associations between clusters. However, it aims to estimate the approximate number of clusters to facilitate further discussion.

Topic dendrogram.

The research stream of AI in healthcare is divided into two main strands. The blue strand focuses on medical information systems and the internet. Some papers are related to healthcare organisations, such as the Internet of Things, meaning that healthcare organisations use AI to support health services management and data analysis. AI applications are also used to improve diagnostic and therapeutic accuracy and the overall clinical treatment process [ 2 ]. If we consider the second block, the red one, three different clusters highlight separate aspects of the topic. The first could be explained as AI and ML predictive algorithms. Through AI applications, it is possible to obtain a predictive approach that can ensure that patients are better monitored. This also allows a better understanding of risk perception for doctors and medical researchers. In the second cluster, the most frequent words are decisions , information system , and support system . This means that AI applications can support doctors and medical researchers in decision-making. Information coming from AI technologies can be used to consider difficult problems and support a more straightforward and rapid decision-making process. In the third cluster, it is vital to highlight that the ML model can deal with vast amounts of data. From those inputs, it can return outcomes that can optimise the work of healthcare organisations and scheduling of medical activities.

Furthermore, the word cloud in Fig.  8 highlights aspects of AI in healthcare, such as decision support systems, decision-making, health services management, learning systems, ML techniques and diseases. The figure depicts how AI is linked to healthcare and how it is used in medicine.

Word cloud.

Figure  9 represents the search trends based on the keywords analysed. The research started in 2012. First, it identified research topics related to clinical decision support systems. This topic was recurrent during the following years. Interestingly, in 2018, studies investigated AI and natural language processes as possible tools to manage patients and administrative elements. Finally, a new research stream considers AI's role in fighting COVID-19 [ 65 , 66 ].

Keywords frequency.

Table  9 represents the number of citations from other articles within the top 20 rankings. The analysis allows the benchmark studies in the field to be identified [ 48 ]. For instance, Burke et al. [ 67 ] writes the most cited paper and analyses efficient nurse rostering methodologies. The paper critically evaluates tangible interdisciplinary solutions that also include AI. Immediately thereafter, Ahmed M.A.'s article proposes a data-driven optimisation methodology to determine the optimal number of healthcare staff to optimise patients' productivity [ 68 ]. Finally, the third most cited article lays the groundwork for developing deep learning by considering diverse health and administrative information [ 51 ].

This section analyses the diffusion of AI in healthcare around the world. It highlights countries to show the geographies of this research. It includes all published articles, the total number of citations, and the collaboration network. The following sub-sections start with an analysis of the total number of published articles.

Country total articles

Figure  9 and Table  10 display the countries where AI in healthcare has been considered. The USA tops the list of countries with the maximum number of articles on the topic (215). It is followed by China (83), the UK (54), India (51), Australia (54), and Canada (32). It is immediately evident that the theme has developed on different continents, highlighting a growing interest in AI in healthcare. The figure shows that many areas, such as Russia, Eastern Europe and Africa except for Algeria, Egypt, and Morocco, have still not engaged in this scientific debate.

Country publications and collaboration map

This section discusses articles on AI in healthcare in terms of single or multiple publications in each country. It also aims to observe collaboration and networking between countries. Table  11 and Fig.  10 highlight the average citations by state and show that the UK, the USA, and Kuwait have a higher average number of citations than other countries. Italy, Spain and New Zealand have the most significant number of citations.

Articles per country.

Figure  11 depicts global collaborations. The blue colour on the map represents research cooperation among nations. Additionally, the pink border linking states indicates the extent of collaboration between authors. The primary cooperation between nations is between the USA and China, with two collaborative articles. Other collaborations among nations are limited to a few papers.

Collaboration map.

Artificial intelligence for healthcare: applications

This section aims to strengthen the research scope by answering RQ3: What are the research applications of artificial intelligence for healthcare?

Benefiting from the topical dendrogram, researchers will provide a development model based on four relevant variables [ 69 , 70 ]. AI has been a disruptive innovation in healthcare [ 4 ]. With its sophisticated algorithms and several applications, AI has assisted doctors and medical professionals in the domains of health information systems, geocoding health data, epidemic and syndromic surveillance, predictive modelling and decision support, and medical imaging [ 2 , 9 , 10 , 64 ]. Furthermore, the researchers considered the bibliometric analysis to identify four macro-variables dominant in the field and used them as authors' keywords. Therefore, the following sub-sections aim to explain the debate on applications in healthcare for AI techniques. These elements are shown in Fig.  12 .

Dominant variables for AI in healthcare.

Health services management

One of the notable aspects of AI techniques is potential support for comprehensive health services management. These applications can support doctors, nurses and administrators in their work. For instance, an AI system can provide health professionals with constant, possibly real-time medical information updates from various sources, including journals, textbooks, and clinical practices [ 2 , 10 ]. These applications' strength is becoming even more critical in the COVID-19 period, during which information exchange is continually needed to properly manage the pandemic worldwide [ 71 ]. Other applications involve coordinating information tools for patients and enabling appropriate inferences for health risk alerts and health outcome prediction [ 72 ]. AI applications allow, for example, hospitals and all health services to work more efficiently for the following reasons:

Clinicians can access data immediately when they need it.

Nurses can ensure better patient safety while administering medication.

Patients can stay informed and engaged in their care by communicating with their medical teams during hospital stays.

Additionally, AI can contribute to optimising logistics processes, for instance, realising drugs and equipment in a just-in-time supply system based totally on predictive algorithms [ 73 , 74 ]. Interesting applications can also support the training of personnel working in health services. This evidence could be helpful in bridging the gap between urban and rural health services [ 75 ]. Finally, health services management could benefit from AI to leverage the multiplicity of data in electronic health records by predicting data heterogeneity across hospitals and outpatient clinics, checking for outliers, performing clinical tests on the data, unifying patient representation, improving future models that can predict diagnostic tests and analyses, and creating transparency with benchmark data for analysing services delivered [ 51 , 76 ].

Predictive medicine

Another relevant topic is AI applications for disease prediction and diagnosis treatment, outcome prediction and prognosis evaluation [ 72 , 77 ]. Because AI can identify meaningful relationships in raw data, it can support diagnostic, treatment and prediction outcomes in many medical situations [ 64 ]. It allows medical professionals to embrace the proactive management of disease onset. Additionally, predictions are possible for identifying risk factors and drivers for each patient to help target healthcare interventions for better outcomes [ 3 ]. AI techniques can also help design and develop new drugs, monitor patients and personalise patient treatment plans [ 78 ]. Doctors benefit from having more time and concise data to make better patient decisions. Automatic learning through AI could disrupt medicine, allowing prediction models to be created for drugs and exams that monitor patients over their whole lives [ 79 ].

• Clinical decision-making

One of the keyword analysis main topics is that AI applications could support doctors and medical researchers in the clinical decision-making process. According to Jiang et al. [ 64 ], AI can help physicians make better clinical decisions or even replace human judgement in healthcare-specific functional areas. According to Bennett and Hauser [ 80 ], algorithms can benefit clinical decisions by accelerating the process and the amount of care provided, positively impacting the cost of health services. Therefore, AI technologies can support medical professionals in their activities and simplify their jobs [ 4 ]. Finally, as Redondo and Sandoval [ 81 ] find, algorithmic platforms can provide virtual assistance to help doctors understand the semantics of language and learning to solve business process queries as a human being would.

Patient data and diagnostics

Another challenging topic related to AI applications is patient data and diagnostics. AI techniques can help medical researchers deal with the vast amount of data from patients (i.e., medical big data ). AI systems can manage data generated from clinical activities, such as screening, diagnosis, and treatment assignment. In this way, health personnel can learn similar subjects and associations between subject features and outcomes of interest [ 64 ].

These technologies can analyse raw data and provide helpful insights that can be used in patient treatments. They can help doctors in the diagnostic process; for example, to realise a high-speed body scan, it will be simpler to have an overall patient condition image. Then, AI technology can recreate a 3D mapping solution of a patient’s body.

In terms of data, interesting research perspectives are emerging. For instance, we observed the emergence of a stream of research on patient data management and protection related to AI applications [ 82 ].

For diagnostics, AI techniques can make a difference in rehabilitation therapy and surgery. Numerous robots have been designed to support and manage such tasks. Rehabilitation robots physically support and guide, for example, a patient’s limb during motor therapy [ 83 ]. For surgery, AI has a vast opportunity to transform surgical robotics through devices that can perform semi-automated surgical tasks with increasing efficiency. The final aim of this technology is to automate procedures to negate human error while maintaining a high level of accuracy and precision [ 84 ]. Finally, the -19 period has led to increased remote patient diagnostics through telemedicine that enables remote observation of patients and provides physicians and nurses with support tools [ 66 , 85 , 86 ].

This study aims to provide a bibliometric analysis of publications on AI in healthcare, focusing on accounting, business and management, decision sciences and health profession studies. Using the SLR method of Massaro et al. [ 11 ], we provide a reliable and replicable research protocol for future studies in this field. Additionally, we investigate the trend of scientific publications on the subject, unexplored information, future directions, and implications using the science mapping workflow. Our analysis provides interesting insights.

In terms of bibliometric variables, the four leading journals, Journal of Medical Systems , Studies in Health Technology and Informatics , IEEE Journal of Biomedical and Health Informatics , and Decision Support Systems , are optimal locations for the publication of scientific articles on this topic. These journals deal mainly with healthcare, medical information systems, and applications such as cloud computing, machine learning, and AI. Additionally, in terms of h-index, Bushko R.G. and Liu C. are the most productive and impactful authors in this research stream. Burke et al.’s [ 67 ] contribution is the most cited with an analysis of nurse rostering using new technologies such as AI. Finally, in terms of keywords, co-occurrence reveals some interesting insights. For instance, researchers have found that AI has a role in diagnostic accuracy and helps in the analysis of health data by comparing thousands of medical records, experiencing automatic learning with clinical alerts, efficient management of health services and places of care, and the possibility of reconstructing patient history using these data.

Second, this paper finds five cluster analyses in healthcare applications: health services management, predictive medicine, patient data, diagnostics, and finally, clinical decision-making. These technologies can also contribute to optimising logistics processes in health services and allowing a better allocation of resources.

Third, the authors analysing the research findings and the issues under discussion strongly support AI's role in decision support. These applications, however, are demonstrated by creating a direct link to data quality management and the technology awareness of health personnel [ 87 ].

The importance of data quality for the decision-making process

Several authors have analysed AI in the healthcare research stream, but in this case, the authors focus on other literature that includes business and decision-making processes. In this regard, the analysis of the search flow reveals a double view of the literature. On the one hand, some contributions belong to the positivist literature and embrace future applications and implications of technology for health service management, data analysis and diagnostics [ 6 , 80 , 88 ]. On the other hand, some investigations also aim to understand the darker sides of technology and its impact. For example, as Carter [ 89 ] states, the impact of AI is multi-sectoral; its development, however, calls for action to protect personal data. Similarly, Davenport and Kalakota [ 77 ] focus on the ethical implications of using AI in healthcare. According to the authors, intelligent machines raise issues of accountability, transparency, and permission, especially in automated communication with patients. Our analysis does not indicate a marked strand of the literature; therefore, we argue that the discussion of elements such as the transparency of technology for patients is essential for the development of AI applications.

A large part of our results shows that, at the application level, AI can be used to improve medical support for patients (Fig.  11 ) [ 64 , 82 ]. However, we believe that, as indicated by Kalis et al. [ 90 ] on the pages of Harvard Business Review, the management of costly back-office problems should also be addressed.

The potential of algorithms includes data analysis. There is an immense quantity of data accessible now, which carries the possibility of providing information about a wide variety of medical and healthcare activities [ 91 ]. With the advent of modern computational methods, computer learning and AI techniques, there are numerous possibilities [ 79 , 83 , 84 ]. For example, AI makes it easier to turn data into concrete and actionable observations to improve decision-making, deliver high-quality patient treatment, adapt to real-time emergencies, and save more lives on the clinical front. In addition, AI makes it easier to leverage capital to develop systems and facilities and reduce expenses at the organisational level [ 78 ]. Studying contributions to the topic, we noticed that data accuracy was included in the debate, indicating that a high standard of data will benefit decision-making practitioners [ 38 , 77 ]. AI techniques are an essential instrument for studying data and the extraction of medical insight, and they may assist medical researchers in their practices. Using computational tools, healthcare stakeholders may leverage the power of data not only to evaluate past data ( descriptive analytics ) but also to forecast potential outcomes ( predictive analytics ) and to define the best actions for the present scenario ( prescriptive analytics ) [ 78 ]. The current abundance of evidence makes it easier to provide a broad view of patient health; doctors should have access to the correct details at the right time and location to provide the proper treatment [ 92 ].

Will medical technology de-skill doctors?

Further reflection concerns the skills of doctors. Studies have shown that healthcare personnel are progressively being exposed to technology for different purposes, such as collecting patient records or diagnosis [ 71 ]. This is demonstrated by the keywords (Fig.  6 ) that focus on technology and the role of decision-making with new innovative tools. In addition, the discussion expands with Lu [ 93 ], which indicates that the excessive use of technology could hinder doctors’ skills and clinical procedures' expansion. Among the main issues arising from the literature is the possible de-skilling of healthcare staff due to reduced autonomy in decision-making concerning patients [ 94 ]. Therefore, the challenges and discussion we uncovered in Fig.  11 are expanded by also considering the ethical implications of technology and the role of skills.

Implications

Our analysis also has multiple theoretical and practical implications.

In terms of theoretical contribution, this paper extends the previous results of Connelly et al., dos Santos et al, Hao et al., Huang et al., Liao et al. and Tran et al. [ 2 , 19 , 20 , 21 , 22 , 24 ] in considering AI in terms of clinical decision-making and data management quality.

In terms of practical implications, this paper aims to create a fruitful discussion with healthcare professionals and administrative staff on how AI can be at their service to increase work quality. Furthermore, this investigation offers a broad comprehension of bibliometric variables of AI techniques in healthcare. It can contribute to advancing scientific research in this field.

Limitations

Like any other, our study has some limitations that could be addressed by more in-depth future studies. For example, using only one research database, such as Scopus, could be limiting. Further analysis could also investigate the PubMed, IEEE, and Web of Science databases individually and holistically, especially the health parts. Then, the use of search terms such as "Artificial Intelligence" OR "AI" AND "Healthcare" could be too general and exclude interesting studies. Moreover, although we analysed 288 peer-reviewed scientific papers, because the new research topic is new, the analysis of conference papers could return interesting results for future researchers. Additionally, as this is a young research area, the analysis will be subject to recurrent obsolescence as multiple new research investigations are published. Finally, although bibliometric analysis has limited the subjectivity of the analysis [ 15 ], the verification of recurring themes could lead to different results by indicating areas of significant interest not listed here.

Future research avenues

Concerning future research perspectives, researchers believe that an analysis of the overall amount that a healthcare organisation should pay for AI technologies could be helpful. If these technologies are essential for health services management and patient treatment, governments should invest and contribute to healthcare organisations' modernisation. New investment funds could be made available in the healthcare world, as in the European case with the Next Generation EU programme or national investment programmes [ 95 ]. Additionally, this should happen especially in the poorest countries around the world, where there is a lack of infrastructure and services related to health and medicine [ 96 ]. On the other hand, it might be interesting to evaluate additional profits generated by healthcare organisations with AI technologies compared to those that do not use such technologies.

Further analysis could also identify why some parts of the world have not conducted studies in this area. It would be helpful to carry out a comparative analysis between countries active in this research field and countries that are not currently involved. It would make it possible to identify variables affecting AI technologies' presence or absence in healthcare organisations. The results of collaboration between countries also present future researchers with the challenge of greater exchanges between researchers and professionals. Therefore, further research could investigate the difference in vision between professionals and academics.

In the accounting, business, and management research area, there is currently a lack of quantitative analysis of the costs and profits generated by healthcare organisations that use AI technologies. Therefore, research in this direction could further increase our understanding of the topic and the number of healthcare organisations that can access technologies based on AI. Finally, as suggested in the discussion section, more interdisciplinary studies are needed to strengthen AI links with data quality management and AI and ethics considerations in healthcare.

In pursuing the philosophy of Massaro et al.’s [ 11 ] methodological article, we have climbed on the shoulders of giants, hoping to provide a bird's-eye view of the AI literature in healthcare. We performed this study with a bibliometric analysis aimed at discovering authors, countries of publication and collaboration, and keywords and themes. We found a fast-growing, multi-disciplinary stream of research that is attracting an increasing number of authors.

The research, therefore, adopts a quantitative approach to the analysis of bibliometric variables and a qualitative approach to the study of recurring keywords, which has allowed us to demonstrate strands of literature that are not purely positive. There are currently some limitations that will affect future research potential, especially in ethics, data governance and the competencies of the health workforce.

Availability of data and materials

All the data are retrieved from public scientific platforms.

Tagliaferri SD, Angelova M, Zhao X, Owen PJ, Miller CT, Wilkin T, et al. Artificial intelligence to improve back pain outcomes and lessons learnt from clinical classification approaches: three systematic reviews. NPJ Digit Med. 2020;3(1):1–16.

Article   Google Scholar  

Tran BX, Vu GT, Ha GH, Vuong Q-H, Ho M-T, Vuong T-T, et al. Global evolution of research in artificial intelligence in health and medicine: a bibliometric study. J Clin Med. 2019;8(3):360.

Article   PubMed Central   Google Scholar  

Hamid S. The opportunities and risks of artificial intelligence in medicine and healthcare [Internet]. 2016 [cited 2020 May 29]. http://www.cuspe.org/wp-content/uploads/2016/09/Hamid_2016.pdf

Panch T, Szolovits P, Atun R. Artificial intelligence, machine learning and health systems. J Glob Health. 2018;8(2):020303.

Article   PubMed   PubMed Central   Google Scholar  

Yang X, Wang Y, Byrne R, Schneider G, Yang S. Concepts of artificial intelligence for computer-assisted drug discovery | chemical reviews. Chem Rev. 2019;119(18):10520–94.

Article   CAS   PubMed   Google Scholar  

Burton RJ, Albur M, Eberl M, Cuff SM. Using artificial intelligence to reduce diagnostic workload without compromising detection of urinary tract infections. BMC Med Inform Decis Mak. 2019;19(1):171.

Meskò B, Drobni Z, Bényei E, Gergely B, Gyorffy Z. Digital health is a cultural transformation of traditional healthcare. Mhealth. 2017;3:38.

Cho B-J, Choi YJ, Lee M-J, Kim JH, Son G-H, Park S-H, et al. Classification of cervical neoplasms on colposcopic photography using deep learning. Sci Rep. 2020;10(1):13652.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Doyle OM, Leavitt N, Rigg JA. Finding undiagnosed patients with hepatitis C infection: an application of artificial intelligence to patient claims data. Sci Rep. 2020;10(1):10521.

Shortliffe EH, Sepúlveda MJ. Clinical decision support in the era of artificial intelligence. JAMA. 2018;320(21):2199–200.

Article   PubMed   Google Scholar  

Massaro M, Dumay J, Guthrie J. On the shoulders of giants: undertaking a structured literature review in accounting. Account Auditing Account J. 2016;29(5):767–801.

Junquera B, Mitre M. Value of bibliometric analysis for research policy: a case study of Spanish research into innovation and technology management. Scientometrics. 2007;71(3):443–54.

Casadesus-Masanell R, Ricart JE. How to design a winning business model. Harvard Business Review [Internet]. 2011 Jan 1 [cited 2020 Jan 8]. https://hbr.org/2011/01/how-to-design-a-winning-business-model

Aria M, Cuccurullo C. bibliometrix: an R-tool for comprehensive science mapping analysis. J Informetr. 2017;11(4):959–75.

Zupic I, Čater T. Bibliometric methods in management and organization. Organ Res Methods. 2015;1(18):429–72.

Secinaro S, Calandra D. Halal food: structured literature review and research agenda. Br Food J. 2020. https://doi.org/10.1108/BFJ-03-2020-0234 .

Rialp A, Merigó JM, Cancino CA, Urbano D. Twenty-five years (1992–2016) of the international business review: a bibliometric overview. Int Bus Rev. 2019;28(6):101587.

Zhao L, Dai T, Qiao Z, Sun P, Hao J, Yang Y. Application of artificial intelligence to wastewater treatment: a bibliometric analysis and systematic review of technology, economy, management, and wastewater reuse. Process Saf Environ Prot. 2020;1(133):169–82.

Article   CAS   Google Scholar  

Huang Y, Huang Q, Ali S, Zhai X, Bi X, Liu R. Rehabilitation using virtual reality technology: a bibliometric analysis, 1996–2015. Scientometrics. 2016;109(3):1547–59.

Hao T, Chen X, Li G, Yan J. A bibliometric analysis of text mining in medical research. Soft Comput. 2018;22(23):7875–92.

dos Santos BS, Steiner MTA, Fenerich AT, Lima RHP. Data mining and machine learning techniques applied to public health problems: a bibliometric analysis from 2009 to 2018. Comput Ind Eng. 2019;1(138):106120.

Liao H, Tang M, Luo L, Li C, Chiclana F, Zeng X-J. A bibliometric analysis and visualization of medical big data research. Sustainability. 2018;10(1):166.

Choudhury A, Renjilian E, Asan O. Use of machine learning in geriatric clinical care for chronic diseases: a systematic literature review. JAMIA Open. 2020;3(3):459–71.

Connelly TM, Malik Z, Sehgal R, Byrnes G, Coffey JC, Peirce C. The 100 most influential manuscripts in robotic surgery: a bibliometric analysis. J Robot Surg. 2020;14(1):155–65.

Guo Y, Hao Z, Zhao S, Gong J, Yang F. Artificial intelligence in health care: bibliometric analysis. J Med Internet Res. 2020;22(7):e18228.

Choudhury A, Asan O. Role of artificial intelligence in patient safety outcomes: systematic literature review. JMIR Med Inform. 2020;8(7):e18599.

Forliano C, De Bernardi P, Yahiaoui D. Entrepreneurial universities: a bibliometric analysis within the business and management domains. Technol Forecast Soc Change. 2021;1(165):120522.

Secundo G, Del Vecchio P, Mele G. Social media for entrepreneurship: myth or reality? A structured literature review and a future research agenda. Int J Entrep Behav Res. 2020;27(1):149–77.

Dal Mas F, Massaro M, Lombardi R, Garlatti A. From output to outcome measures in the public sector: a structured literature review. Int J Organ Anal. 2019;27(5):1631–56.

Google Scholar  

Baima G, Forliano C, Santoro G, Vrontis D. Intellectual capital and business model: a systematic literature review to explore their linkages. J Intellect Cap. 2020. https://doi.org/10.1108/JIC-02-2020-0055 .

Dumay J, Guthrie J, Puntillo P. IC and public sector: a structured literature review. J Intellect Cap. 2015;16(2):267–84.

Dal Mas F, Garcia-Perez A, Sousa MJ, Lopes da Costa R, Cobianchi L. Knowledge translation in the healthcare sector. A structured literature review. Electron J Knowl Manag. 2020;18(3):198–211.

Mas FD, Massaro M, Lombardi R, Biancuzzi H. La performance nel settore pubblico tra misure di out-put e di outcome. Una revisione strutturata della letteratura ejvcbp. 2020;1(3):16–29.

Dumay J, Cai L. A review and critique of content analysis as a methodology for inquiring into IC disclosure. J Intellect Cap. 2014;15(2):264–90.

Haleem A, Javaid M, Khan IH. Current status and applications of Artificial Intelligence (AI) in medical field: an overview. Curr Med Res Pract. 2019;9(6):231–7.

Paul J, Criado AR. The art of writing literature review: what do we know and what do we need to know? Int Bus Rev. 2020;29(4):101717.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.

Biancone PP, Secinaro S, Brescia V, Calandra D. Data quality methods and applications in health care system: a systematic literature review. Int J Bus Manag. 2019;14(4):p35.

Secinaro S, Brescia V, Calandra D, Verardi GP, Bert F. The use of micafungin in neonates and children: a systematic review. ejvcbp. 2020;1(1):100–14.

Bert F, Gualano MR, Biancone P, Brescia V, Camussi E, Martorana M, et al. HIV screening in pregnant women: a systematic review of cost-effectiveness studies. Int J Health Plann Manag. 2018;33(1):31–50.

Levy Y, Ellis TJ. A systems approach to conduct an effective literature review in support of information systems research. Inf Sci Int J Emerg Transdiscipl. 2006;9:181–212.

Chen G, Xiao L. Selecting publication keywords for domain analysis in bibliometrics: a comparison of three methods. J Informet. 2016;10(1):212–23.

Falagas ME, Pitsouni EI, Malietzis GA, Pappas G. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB J. 2007;22(2):338–42.

Article   PubMed   CAS   Google Scholar  

Sicilia M-A, Garcìa-Barriocanal E, Sànchez-Alonso S. Community curation in open dataset repositories: insights from zenodo. Procedia Comput Sci. 2017;1(106):54–60.

Secinaro S, Calandra D, Secinaro A, Muthurangu V, Biancone P. Artificial Intelligence for healthcare with a business, management and accounting, decision sciences, and health professions focus [Internet]. Zenodo; 2021 [cited 2021 Mar 7]. https://zenodo.org/record/4587618#.YEScpl1KiWh .

Elango B, Rajendran D. Authorship trends and collaboration pattern in the marine sciences literature: a scientometric Study. Int J Inf Dissem Technol. 2012;1(2):166–9.

Jacoby WG. Electoral inquiry section Loess: a nonparametric, graphical tool for depicting relationships between variables q. In 2000.

Andrews JE. An author co-citation analysis of medical informatics. J Med Libr Assoc. 2003;91(1):47–56.

PubMed   PubMed Central   Google Scholar  

White HD, Griffith BC. Author cocitation: a literature measure of intellectual structure. J Am Soc Inf Sci. 1981;32(3):163–71.

Santosh KC. AI-driven tools for coronavirus outbreak: need of active learning and cross-population train/test models on multitudinal/multimodal data. J Med Syst. 2020;44(5):93.

Shickel B, Tighe PJ, Bihorac A, Rashidi P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J Biomed Health Inform. 2018;22(5):1589–604.

Baig MM, GholamHosseini H, Moqeem AA, Mirza F, Lindén M. A systematic review of wearable patient monitoring systems—current challenges and opportunities for clinical adoption. J Med Syst. 2017;41(7):115.

Kumar S, Kumar S. Collaboration in research productivity in oil seed research institutes of India. In: Proceedings of fourth international conference on webometrics, informetrics and scientometrics. p. 28–1; 2008.

Gatto A, Drago C. A taxonomy of energy resilience. Energy Policy. 2020;136:111007.

Levitt JM, Thelwall M. Alphabetization and the skewing of first authorship towards last names early in the alphabet. J Informet. 2013;7(3):575–82.

Saad G. Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics. 2006;69(1):117–20.

Egghe L. Theory and practise of the g-index. Scientometrics. 2006;69(1):131–52.

Schreiber M. A modification of the h-index: the hm-index accounts for multi-authored manuscripts. J Informet. 2008;2(3):211–6.

Engqvist L, Frommen JG. The h-index and self-citations. Trends Ecol Evol. 2008;23(5):250–2.

London School of Economics. 3: key measures of academic influence [Internet]. Impact of social sciences. 2010 [cited 2021 Jan 13]. https://blogs.lse.ac.uk/impactofsocialsciences/the-handbook/chapter-3-key-measures-of-academic-influence/ .

Lotka A. The frequency distribution of scientific productivity. J Wash Acad Sci. 1926;16(12):317–24.

Khan G, Wood J. Information technology management domain: emerging themes and keyword analysis. Scientometrics. 2015;9:105.

Oxford University Press. Oxford English Dictionary [Internet]. 2020. https://www.oed.com/ .

Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. 2017;2(4):230–43.

Calandra D, Favareto M. Artificial Intelligence to fight COVID-19 outbreak impact: an overview. Eur J Soc Impact Circ Econ. 2020;1(3):84–104.

Bokolo Anthony Jnr. Use of telemedicine and virtual care for remote treatment in response to COVID-19 pandemic. J Med Syst. 2020;44(7):132.

Burke EK, De Causmaecker P, Berghe GV, Van Landeghem H. The state of the art of nurse rostering. J Sched. 2004;7(6):441–99.

Ahmed MA, Alkhamis TM. Simulation optimization for an emergency department healthcare unit in Kuwait. Eur J Oper Res. 2009;198(3):936–42.

Forina M, Armanino C, Raggio V. Clustering with dendrograms on interpretation variables. Anal Chim Acta. 2002;454(1):13–9.

Wartena C, Brussee R. Topic detection by clustering keywords. In: 2008 19th international workshop on database and expert systems applications. 2008. p. 54–8.

Hussain AA, Bouachir O, Al-Turjman F, Aloqaily M. AI Techniques for COVID-19. IEEE Access. 2020;8:128776–95.

Agrawal A, Gans JS, Goldfarb A. Exploring the impact of artificial intelligence: prediction versus judgment. Inf Econ Policy. 2019;1(47):1–6.

Chakradhar S. Predictable response: finding optimal drugs and doses using artificial intelligence. Nat Med. 2017;23(11):1244–7.

Fleming N. How artificial intelligence is changing drug discovery. Nature. 2018;557(7707):S55–7.

Guo J, Li B. The application of medical artificial intelligence technology in rural areas of developing countries. Health Equity. 2018;2(1):174–81.

Aisyah M, Cockcroft S. A snapshot of data quality issues in Indonesian community health. Int J Netw Virtual Organ. 2014;14(3):280–97.

Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J. 2019;6(2):94–8.

Mehta N, Pandit A, Shukla S. Transforming healthcare with big data analytics and artificial intelligence: a systematic mapping study. J Biomed Inform. 2019;1(100):103311.

Collins GS, Moons KGM. Reporting of artificial intelligence prediction models. Lancet. 2019;393(10181):1577–9.

Bennett CC, Hauser K. Artificial intelligence framework for simulating clinical decision-making: a Markov decision process approach. Artif Intell Med. 2013;57(1):9–19.

Redondo T, Sandoval AM. Text Analytics: the convergence of big data and artificial intelligence. Int J Interact Multimed Artif Intell. 2016;3. https://www.ijimai.org/journal/bibcite/reference/2540 .

Winter JS, Davidson E. Big data governance of personal health information and challenges to contextual integrity. Inf Soc. 2019;35(1):36–51.

Novak D, Riener R. Control strategies and artificial intelligence in rehabilitation robotics. AI Mag. 2015;36(4):23–33.

Tarassoli SP. Artificial intelligence, regenerative surgery, robotics? What is realistic for the future of surgery? Ann Med Surg (Lond). 2019;17(41):53–5.

Saha SK, Fernando B, Cuadros J, Xiao D, Kanagasingam Y. Automated quality assessment of colour fundus images for diabetic retinopathy screening in telemedicine. J Digit Imaging. 2018;31(6):869–78.

Gu D, Li T, Wang X, Yang X, Yu Z. Visualizing the intellectual structure and evolution of electronic health and telemedicine research. Int J Med Inform. 2019;130:103947.

Madnick S, Wang R, Lee Y, Zhu H. Overview and framework for data and information quality research. J Data Inf Qual. 2009;1:1.

Chen X, Liu Z, Wei L, Yan J, Hao T, Ding R. A comparative quantitative study of utilizing artificial intelligence on electronic health records in the USA and China during 2008–2017. BMC Med Inform Decis Mak. 2018;18(5):117.

Carter D. How real is the impact of artificial intelligence? Bus Inf Surv. 2018;35(3):99–115.

Kalis B, Collier M, Fu R. 10 Promising AI Applications in Health Care. 2018;5.

Biancone P, Secinaro S, Brescia V, Calandra D. Management of open innovation in healthcare for cost accounting using EHR. J Open Innov Technol Market Complex. 2019;5(4):99.

Kayyali B, Knott D, Van Kuiken S. The ‘big data’ revolution in US healthcare [Internet]. McKinsey & Company. 2013 [cited 2020 Aug 14]. https://healthcare.mckinsey.com/big-data-revolution-us-healthcare/ .

Lu J. Will medical technology deskill doctors? Int Educ Stud. 2016;9(7):130–4.

Hoff T. Deskilling and adaptation among primary care physicians using two work innovations. Health Care Manag Rev. 2011;36(4):338–48.

Picek O. Spillover effects from next generation EU. Intereconomics. 2020;55(5):325–31.

Sousa MJ, Dal Mas F, Pesqueira A, Lemos C, Verde JM, Cobianchi L. The potential of AI in health higher education to increase the students’ learning outcomes. TEM J. 2021. ( In press ).

Download references

Acknowledgements

The authors are grateful to the Editor-in-Chief for the suggestions and all the reviewers who spend a part of their time ensuring constructive feedback to our research article.

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Author information

Authors and affiliations.

Department of Management, University of Turin, Turin, Italy

Silvana Secinaro, Davide Calandra & Paolo Biancone

Ospedale Pediatrico Bambino Gesù, Rome, Italy

Aurelio Secinaro

Institute of Child Health, University College London, London, UK

Vivek Muthurangu

You can also search for this author in PubMed   Google Scholar

Contributions

SS and PB, Supervision; Validation, writing, AS and VM; Formal analysis, DC and AS; Methodology, DC; Writing; DC, SS and AS; conceptualization, VM, PB; validation, VM, PB. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Davide Calandra .

Ethics declarations

Ethical approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Secinaro, S., Calandra, D., Secinaro, A. et al. The role of artificial intelligence in healthcare: a structured literature review. BMC Med Inform Decis Mak 21 , 125 (2021). https://doi.org/10.1186/s12911-021-01488-9

Download citation

Received : 24 December 2020

Accepted : 01 April 2021

Published : 10 April 2021

DOI : https://doi.org/10.1186/s12911-021-01488-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Patient data

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

• General enquiries: [email protected]

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• For authors
• Browse by collection
• BMJ Journals

You are here

• Volume 14, Issue 8
• Evidence-informed, experience-based co-design: a novel framework integrating research evidence and lived experience in priority-setting and co-design of health services
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0001-9721-9225 Claire Morley 1 ,
• Kim Jose 1 ,
• Sonj E Hall 2 ,
• Kelly Shaw 1 , 3 ,
• Deirdre McGowan 1 ,
• Martina Wyss 4 ,
• http://orcid.org/0000-0002-4112-3491 Tania Winzenberg 1
• 1 Menzies Institute for Medical Research , University of Tasmania , Hobart , Tasmania , Australia
• 2 Bellberry Ltd , Eastwood , South Australia , Australia
• 3 KPHealth , Hobart , Tasmania , Australia
• 4 Primary Health Tasmania , Launceston , Tasmania , Australia
• Correspondence to Dr Claire Morley; claire.morley{at}utas.edu.au

Objective To describe a new co-design framework termed Evidence-informed, Experience-based Co-design (E2CD).

Background Involving consumers and clinicians in planning, designing and implementing services results in the end-product being more likely to meet the needs of end-users and increases the likelihood of their uptake and sustainability. Different forms and definitions of co-design have been described in the literature and have had varying levels of success in health service redesign. However, many fall short of including people with lived experience in all aspects of the process, particularly in setting priorities for service (re)design. In addition, health services need to deliver evidence-based care as well as care that meets the needs of users, yet few ways of integrating research evidence into co-design processes are described. This paper describes a new framework to approach co-design which addresses these issues. We believe that it offers a roadmap to address some of healthcare’s most wicked problems and potentially improve outcomes for some of the most vulnerable people in our society. We use improving services for people with high healthcare service utilisation as a working example of the Framework’s application.

Conclusion Evidence-informed experience-based co-design has the potential to be used as a framework for co-design that integrates research evidence with lived experience and provides people with lived experience a central role in decision-making about prioritising and designing services to meet their needs.

• Health policy
• Organisation of health services
• Patient Participation
• Primary Health Care
• PUBLIC HEALTH

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/bmjopen-2024-084620

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Involving consumers and clinicians in the planning, design and implementation of services results in the end-product being more likely to meet the needs of end-users, 1 2 thereby increasing the livelihood of uptake and sustainability of new services. 3 Co-production is an umbrella term covering an array of evolving methods of including consumers in the design of health and social care services. 4 While definitions and uses of different ‘co’ methods have caused some confusion and much discussion in the academic literature, a recent review has called for a shift in focus away from different definitions and towards advocating for the often similar principles of these methods to be operationalised and translated in practice. 4

Experience-based co-design (EBCD) is one approach to health system (re)design that is founded on the principles of action research and design thinking. 5 The co-design principles of equal partnership, openness, respect, empathy and design together have been suggested by one Australian group as critical to ensure that consumers are considered as equals at all stages of the design process. 6 EBCD has been defined as an approach that enables staff and patients (or other service users) to co-design services and/or care pathways, together in partnership . 7 Developed in the UK, EBCD is described as being about more than simply promoting greater patient participation, but about placing the patients’ experiences at the centre of the design process. 8 The Point of Care Foundation (UK) proposes eight stages in EBCD, namely, observation, interviews of service users, development of a trigger film, service provider feedback, service user feedback, joint service user and provider feedback, co-design groups and celebration event. 7 Bate and Robert state that the storytelling, or identification of ‘touch points’ from the trigger film, is the very basis of experience-based design. 8 However, EBCD in practice has undergone many adaptations, often due to costs and time constraints. 9 10 In many instances, this has resulted in a process commonly called accelerated EBCD, where adaptations include using stock ‘trigger films’, or absence of a celebration event. 11 A recent systematic review of EBCD use highlighted its frequent adaptation due to costs and time constraints. 10 Two phases, each with consumer involvement, were considered crucial, namely, experience gathering and co-design. 10 Additionally, Green et al recommend supplementing the process with service provider experiences, due to the potential differences in priorities and beliefs of users and providers. 10 This review noted that EBCD’s predominant use has been for service improvement in local settings but notes that its use could be extended beyond this, for example, for developing new services or redesign of health policy. 10 However, it is less clear how EBCD can be integrated earlier into setting priorities for health service and policy design.

While health services need to meet the needs of its users, it is imperative that they also deliver evidence-based care. However, researchers commonly drive design processes in healthcare research, for example, undertaking literature reviews prior to engaging with stakeholders. 12 This may narrow the range of potential solutions from the outset or leave teams conflicted as to whether published literature or stakeholder views should be given priority. 12 Indeed, one qualitative evaluation of participants’ experiences of co-design identified the need to limit researcher domination of the co-design process to enable the establishment and maintenance of genuine partnerships. 13 Approaches to using research evidence to supplement user experience in co-design are also infrequently described. One proposed approach to improving commissioning of health services integrates research evidence into a co-creation process in a limited way but does not emphasise the role of people with lived experience in decision-making or the importance of incorporating knowledge from lived experience in co-design. 14 This is a critical gap that needs to be addressed.

People with high healthcare service use (HSU) are the small proportion of people who use a disproportionately high amount of healthcare services, determined by frequency of healthcare use such as hospital admissions, emergency department (ED) presentations, primary care visits or by incurring high healthcare costs. 15 Indeed, there are calls for HSU to be considered a red flag of patients’ physical, mental, spiritual and social deprivation . 16 Healthcare service redesign is a recognised priority for improving services for people with HSU, 17 though not a panacea given how factors outside the health system, such as unemployment and homelessness, contribute to their issues. 18 Interventions to reduce health service use by people with HSU have often been ineffective. Failure to recognise patient needs in intervention design may contribute to this. Reviews highlight the importance of considering patients’ unique needs 19 and contexts. 20 Despite this, the diverse needs of people with HSU are poorly understood. Systematic reviews of interventions are silent on whether and how lived experience has been incorporated into the design of interventions. 20–22 Failing to do this may lead to services being designed that do not address patient needs or barriers to access. 23

This paper aims to describe a new framework building on EBCD to address the limitations of co-design approaches identified above, using the complex problem of HSU as an example to its potential application.

The framework: evidence-informed, experience-based co-design

We have developed a new framework called Evidence-informed, Experience-based Co-design (E2CD) that applies mixed methods research methodologies to integrate research evidence with lived experience in a co-design process in which people with lived experience have a central, decision-making role. Components of the framework are shown in figure 1 , which also shows that it builds on EBCD, 7 retaining its strong emphasis on the primacy of views of people with lived experience and of their involvement in co-design. The process is cyclic and iterative and can begin at any phase depending on the context in which (re)design is needed. Key features that together distinguish it from other approaches are that:

people with lived experience are kept central to the process by their lived experience informing each step and by them being involved in all phases including explicit decision-making roles for priority setting and co-design.

data analysis, lived experience and evidence synthesis are integrated and incorporated throughout the process, without a priori decisions made by the researchers.

researchers are responsive to participants’ needs and generate evidence as requested, thereby acting as equal partners in rather than drivers of the process. Throughout, participants are encouraged to voice questions that can be addressed by rapid, focused syntheses of existing evidence.

it has a priority-setting phase that gives people with lived experience a role in deciding co-design priorities.

services and their evaluations are co-designed in one contiguous process.

it is applicable to both the design of new services and redesign of existing services.

it is iterative. Phases can be revisited when more information emerges.

• Download figure
• Open in new tab
• Download powerpoint

Graphic depiction of the E vidence-informed, E xperience-based C o-design (E2CD) Framework. The process moves iteratively through 5 phases, with a focus on people with lived experience at all times as shown by their central placement in the Figure. Participants at any time can call on rapid responsive focused evidence synthesis to support the process as shown by this surrounding the process (outer circle of figure). In the understanding and refining phases, holistic understanding of the issue being examined from consumer, clinician, and policy-maker perspectives is sought. A formal process is then undertaken to develop a person-centred, stakeholder-informed, and evidence-based list of priority areas for new and/or improved models of care and service (Prioritising phase). The highest priority area is then addressed using an EBCD-based approach to co-design a new, or re-design an existing model of service delivery and its evaluation (Service and evaluation co-design phase). The improvements are implemented and evaluated with results from the Implementation and Evaluation phase informing service development for the next priority area.

Evidence-informed, experience-based co-design in action

We are piloting the application of the E2CD framework to the issue of HSU in our local context in Tasmania. Tasmania is Australia’s only island state and has a population that is older, more geographically dispersed and more socioeconomically disadvantaged than the rest of Australia, 24 25 which magnifies the effects of HSU on access and availability of healthcare services. We use the issue of HSU as an example to illustrate how the E2CD framework could be applied (illustrated in figure 2 ).

Application of Evidence-informed, experience-based co-design (E2CD) to high healthcare service utilisation (HSU).

Understanding

The understanding phase has three components: understanding lived experience, using available data to understand HSU and including lessons from evaluations of initiatives already in place. To do this, we are:

evaluating a new, co-designed, nurse-led service that offers wrap-around care for people who have experienced ≥4 hospital admissions in a 12-month period 26 ;

undertaking a data linkage study that links Tasmanian public hospital admission, ED presentation and cause of death data to enable a comprehensive understanding of the characteristics and patterns of different manifestations of HSU locally, and;

conducting a qualitative study interviewing Tasmanians with lived experience with different types of HSU to gain an in-depth understanding of the causes, consequences and needs of people with HSU in Tasmania ( n =60 approx.). We will use purposive sampling to ensure a diverse range of people are included (eg, by age, sex, rurality and region).

In the refining phase, approximately 20 policymakers, clinicians (acute and primary/community care), commissioners of services, funders and other stakeholders will be interviewed to gain their perspectives on aspects of HSU such as its causes, consequences, priority areas and potential solutions. This will augment information from the understanding phase. Although using focus groups to gather participant experience is recommended in EBCD, 7 we have chosen to undertake interviews as EBCD facilitators have reported that individually interviewing service providers has the added benefit of engaging and enhancing their commitment to the process. 27 Service mapping will also be undertaken to identify services already available to people with HSU.

Prioritising

In the prioritising phase, a facilitated workshop with membership from all stakeholder groups previously involved including people with lived experience will be held to determine local priorities. This may be a service use pattern in one area (eg, ED presentations in a particular region) or a particular condition (eg, diabetes). We plan to use Nominal Group Technique 28 as this is a consensus method used to avoid domination of decision-making by individuals who have a vested interest in the outcome 29 and so aligns with the goals of E2CD.

Service evaluation and co-design

During this phase, a group, including members from all stakeholder groups involved thus far, will be convened to co-design/redesign a service to address the agreed highest priority. If needed, interviews with people who are potential end-users of that specific service will be undertaken, ensuring that potential end-users are central to the process. Evidence from all preceding phases will be communicated to the design group. EBCD principles will be used 7 to ensure that the voice of people with lived experience is central to all decisions ( figure 2 ). The evaluation framework for the new service will be co-designed simultaneously.

Implementation and evaluation

Involving all key stakeholders from the outset, including funders and commissioners of services, is intended to facilitate early implementation of the co-designed service. 9 The evaluation of the new service will run concurrently with its implementation, with lessons learnt informing ongoing improvements within the service. This evaluation is important, as there is a reported lack of robust evaluation of interventions that are co-produced, leading to a paucity of evidence that co-designed services lead to improved health outcomes. 30 Once this is implemented, the co-design group can revisit the priority setting phase and select the next priority for another co-design phase.

Rapid, responsive, focused evidence synthesis

To reduce the risk that researchers drive co-design outcomes rather than evidence synthesis being a tool used by people participating in co-design, at any phase of E2CD, stakeholders, including people with lived experience, can identify questions they would like the research team to address. The research team is equipped to rapidly respond to these requests, ensuring that the relevant information is summarised and provided to the stakeholders in a timely manner to support decision-making. Methods such as rapid reviews will be used for this.

With current expenditure on health growing faster than GDP in most countries, 31 it is critical to ensure that expenditure on new initiatives and services is channelled to those that are both evidence-based and acceptable to end-users. Many governments, including Australia, are moving towards ensuring that healthcare is value-based and that the voice of consumers is included at all stages of healthcare planning. 32 Involving all potential end-users in prioritising and co-designing services may provide some assurance of acceptability and thereby sustainability of new initiatives. The E2CD approach we propose provides a framework that systematically involves all stakeholders in service (re)design from priority setting right through to implementation and evaluation, integrates research evidence with lived experience and gives people with lived experience a central, decision-making role. It offers a roadmap to address some of healthcare’s most wicked problems and potentially improve outcomes for some of the most vulnerable people in our society, such as people with HSU. In doing so, it has the potential to be a guide for researchers seeking to improve integration of lived experience into designing and testing complex interventions, and it has potential application in the commissioning of health services. In addition, it has the potential to address the researcher/consumer power imbalance often identified as an issue in health service redesign.

The potential for power imbalances between co-design participants is widely acknowledged. 2 4 5 11 33 34 This is not always considered a negative, with one study finding that co-design can challenge traditional relationships between patients and clinicians and may blur the boundaries between practice and academia. 2 Nonetheless, co-design leaders need to be aware that real and perceived power imbalances can derail the process and ensure no one group dominates. This can be achieved by setting up power-sharing structures and promoting collective ownership. 2 13 33 34 The E2CD framework, in which researchers are equal partners rather than drivers of co-design, aims to avoid ‘researcher dominance’. 13 Having a research team work in partnership to provide evidence synthesis as requested by the co-design participants could help prevent researchers’ a priori judgements of evidence from outweighing the evidence of lived experience and unduly influencing co-design outcomes. It may also lessen the power disparity that is so often an impediment to true partnerships in co-design. 13

The E2CD framework can be applied to healthcare research and could be particularly useful when undertaking research aiming to improve health services or implement complex interventions. Many research funding bodies now strongly encourage or even mandate that researchers engage with consumers when designing research studies. 35 36 They believe that the inclusion of consumer priorities, values and experiences is a means of ensuring research delivers fit-for-purpose outcomes that will be adopted by end-users. 36 The Framework provides a way for researchers to work with people with lived experience and integrate evidence into choosing research topics to address (prioritisation) and the co-design of any interventions/models of care they choose to study. This is likely to result in a greater alignment of research with stakeholder needs and improve the impact of research performed. It is important, however, to ensure that all stakeholders are cognisant of the time, costs and risks associated with using co-design in research. 37 These need to be acknowledged by funding bodies in terms of available funding and timelines and researchers in terms of deciding when to best use this approach.

The commissioning of health services is another area in which the E2CD framework could be useful. Commissioning is defined as a continual and iterative cycle involving the development and implementation of services based on needs assessment, planning, co-design, procurement, monitoring and evaluation 38 and occurs in countries including the UK, New Zealand and Australia. 39 Undertaking a needs analysis with end-user involvement is the basis for sound commissioning decisions. 40 However, consumer involvement in the commissioning process to date has been mainly limited to making changes to minor aspects of service provision rather than any engagement in strategic planning or priority setting. 41 The lack of flexibility in how funds can be expended has been identified as another constraint, as it limits the ability to be innovative with services at a local level. 39 41 42 We believe that the E2CD framework could help address these limitations. First, if adhered to, it provides a blueprint for the incorporation of lived experience and the involvement of end-users, including consumers, at every stage of the commissioning process, ensuring that the expressed needs of local communities are understood by providers and incorporated into service (re)design. Second, it provides a structure for Primary Heath Networks to engage proactively within their communities to identify local needs and generate evidence to lobby for funds to address local priorities.

This article has some limitations. First, E2CD describes an overarching approach rather than a detailed methodology, so when applying it, users will need to choose methods appropriate for each phase. These choices will depend on factors including the context in which E2CD occurs, the health issue it aims to address and any specific barriers to engagement for people with lived experience. Choosing sensitive engagement processes and tailoring methods for priority-setting and co-design to meet the needs of specific groups of people with lived experience will be critical to its successful application. Second, we use HSU as an example to illustrate the potential utility of the E2CD framework. Therefore, a detailed explanation of the specific methods to be used in its application is beyond this article’s scope. Finally, as consumers were not involved in developing the framework, evaluating its use in practice with people with lived experience is essential.

As the cost of healthcare continues to rise, it is crucial that services deliver evidence-based care that meets the needs of its users. Involving all stakeholders in decisions regarding prioritising areas for service provision as well as designing services increases the likelihood that they will be embraced by end-users and outcomes will be sustained. While not a panacea, evidence-informed, experience-based co-design provides a framework to potentially address some of healthcare’s most wicked problems.

Ethics approval

Not applicable.

• Lacombe R , et al
• Grindell C ,
• Croot L , et al
• Luker JA , et al
• Masterson D ,
• Areskoug Josefsson K ,
• Robert G , et al
• Moll S, D ,
• Miatello A ,
• Murray-Leung L , et al
• Agency for Clinical Innovation
• The Point of Care Foundation
• Sangiorgi D ,
• Büscher M , et al
• Teleni L , et al
• Cornwell J ,
• Locock L , et al
• Rousseau N ,
• Turner KM ,
• Duncan E , et al
• Pallesen KS ,
• Anjara S , et al
• van der Borght C ,
• Leithaus M , et al
• Canadian Institute for Health Information
• Goodwin A ,
• Henschen BL ,
• O’Dwyer LC , et al
• Berkman ND ,
• Seibert J , et al
• Delaney RK ,
• Sisco-Taylor B ,
• Fagerlin A , et al
• Chouinard M-C ,
• Lambert M , et al
• Schumacher JR ,
• Hall AG , et al
• Australian Bureau of Statistics
• Tasmanian Medicare Local
• Tasmanian Department of Health
• Tollyfield R
• Rankin NM ,
• McGregor D ,
• Butow PN , et al
• Slattery P ,
• World Health Organisation
• Australian Health Ministers
• McGillion M ,
• Chambers EM , et al
• Dimopoulos-Bick TL ,
• O’Connor C ,
• Montgomery J , et al
• National Institute for Health and Care Research
• Medical Research Future Fund
• Kothari A ,
• Commonwealth Government of Australia
• Harris-Roxas B
• Gardner K ,
• Dickinson H ,
• Worswick L , et al

Contributors CM and TW conceptualised the framework. CM, KJ, SEH, KS, DM, MW and TW provided input into the framework design. CM and TW drafted the paper with input from KJ, SEH, KS, DM and MW. CM, KJ, SEH, KS, DM, MW and TW approved the final version of the paper. TW is the guarantor.

Funding This work is supported by Primary Health Tasmania under the Australian Government’s Primary Health Networks Program and by an Australian Government Research Training Program (RTP) Scholarship - DM, a doctoral candidate, is a recipient of scholarship funding from both entities.

Competing interests None declared.

Provenance and peer review Not commissioned; externally peer reviewed.

Read the full text or download the PDF:

• Fact sheets
• Facts in pictures

Publications

• Questions and answers
• Tools and toolkits
• Endometriosis
• Excessive heat
• Mental disorders
• Polycystic ovary syndrome
• All countries
• Eastern Mediterranean
• South-East Asia
• Western Pacific
• Data by country
• Country presence 
• Country strengthening 
• Country cooperation strategies 
• News releases
• Feature stories
• Press conferences
• Commentaries
• Photo library
• Afghanistan
• Cholera 
• Coronavirus disease (COVID-19)
• Greater Horn of Africa
• Israel and occupied Palestinian territory
• Disease Outbreak News
• Situation reports
• Weekly Epidemiological Record
• Surveillance
• Health emergency appeal
• International Health Regulations
• Independent Oversight and Advisory Committee
• Classifications
• Data collections
• Global Health Estimates
• Mortality Database
• Sustainable Development Goals
• Health Inequality Monitor
• Global Progress
• World Health Statistics
• Partnerships
• Committees and advisory groups
• Collaborating centres
• Technical teams
• Organizational structure
• Initiatives
• General Programme of Work
• WHO Academy
• Investment in WHO
• WHO Foundation
• External audit
• Financial statements
• Internal audit and investigations 
• Programme Budget
• Results reports
• Governing bodies
• World Health Assembly
• Executive Board
• Member States Portal
• Health topics /

Research is indispensable for resolving public health challenges – whether it be tackling diseases of poverty, responding to rise of chronic diseases,  or ensuring that mothers have access to safe delivery practices.

Likewise, shared vulnerability to global threats, such as severe acute respiratory syndrome, Ebola virus disease, Zika virus and avian influenza has mobilized global research efforts in support of enhancing capacity for preparedness and response. Research is strengthening surveillance, rapid diagnostics and development of vaccines and medicines.

Public-private partnerships and other innovative mechanisms for research are concentrating on neglected diseases in order to stimulate the development of vaccines, drugs and diagnostics where market forces alone are insufficient.

Research for health spans 5 generic areas of activity:

• measuring the magnitude and distribution of the health problem;
• understanding the diverse causes or the determinants of the problem, whether they are due to biological, behavioural, social or environmental factors;
• developing solutions or interventions that will help to prevent or mitigate the problem;
• implementing or delivering solutions through policies and programmes; and
• evaluating the impact of these solutions on the level and distribution of the problem.

High-quality research is essential to fulfilling WHO’s mandate for the attainment by all peoples of the highest possible level of health. One of the Organization’s core functions is to set international norms, standards and guidelines, including setting international standards for research.

Under the “WHO strategy on research for health”, the Organization works to identify research priorities, and promote and conduct research with the following 4 goals:

• Capacity - build capacity to strengthen health research systems within Member States.
• Priorities - support the setting of research priorities that meet health needs particularly in low- and middle-income countries.
• Standards - develop an enabling environment for research through the creation of norms and standards for good research practice.
• Translation - ensure quality evidence is turned into affordable health technologies and evidence-informed policy.
• Prequalification of medicines by WHO
• Global Observatory on Health R&D
• Global Observatory on Health Research and Development
• Implementation research toolkit
• Ethics in implementation research: participant's guide
• Ethics in implementation research: facilitator's guide
• Ethics in epidemics, emergencies and disasters: Research, surveillance and patient care: WHO training manual
• WHA58.34 Ministerial Summit on Health Research
• WHA60.15 WHO's role and responsibilities in health research
• WHA63.21 WHO's role and responsibilities in health research
• EB115/30 Ministerial Summit on Health Research: report by the Secretariat
• Science division

WHO consults on action plan for sustainable clinical research infrastructure

WHO advisory group convenes its first meeting on responsible use of the life sciences in Geneva

Challenging harmful masculinities and engaging men and boys in sexual and reproductive health

Stakeholders convene in Uganda on responsible use of the life sciences

Target product profiles for animal plasma-derived antivenoms: antivenoms for treatment of snakebite envenoming...

We describe here WHO public-benefit Target Product Profiles (TPPs) for antivenoms intended to be used for treatment of snakebite envenoming in South Asia,...

WHO global research priorities for sexually transmitted infections

Sexually transmitted infections (STIs) are widespread globally and negatively affect sexual and reproductive health. Gaps in evidence and in available...

Report of the sixth meeting of the WHO Diagnostic Technical Advisory Group for Neglected Tropical Diseases:...

The World Health Organization’s Global Neglected Tropical Diseases Programme (WHO/NTD) manages a diverse portfolio of 21 diseases and disease groups,1 ...

Target product profile for a diagnostic test to confirm cure of visceral leishmaniasis

Leishmaniasis is caused by protozoan parasites which are transmitted by the bite of infected female phlebotomine sandflies. The disease is poverty-related...

Coordinating R&D on antimicrobial resistance

Ensuring responsible use of life sciences research

Prioritizing diseases for research and development in emergency contexts

Promoting research on Buruli ulcer

Research in maternal, perinatal, and adolescent health

Undertaking health law research

Feature story

One year on, Global Observatory on Health R&D identifies striking gaps and inequalities

Video: Open access to health: WHO joins cOAlition S

Video: Multisectional research on sleeping sickness in Tanzania in the context of climate change

Related health topics

Clinical trials

Global health ethics

Health Laws

Intellectual property and trade

Related links

Research and Development Blueprint

WHO Collaborating Centres

R&D Blueprint for Action to Prevent Epidemics

International Clinical Trials Registry Platform

Research Topics & Ideas: Healthcare

100+ Healthcare Research Topic Ideas To Fast-Track Your Project

Finding and choosing a strong research topic is the critical first step when it comes to crafting a high-quality dissertation, thesis or research project. If you’ve landed on this post, chances are you’re looking for a healthcare-related research topic , but aren’t sure where to start. Here, we’ll explore a variety of healthcare-related research ideas and topic thought-starters across a range of healthcare fields, including allopathic and alternative medicine, dentistry, physical therapy, optometry, pharmacology and public health.

NB – This is just the start…

The topic ideation and evaluation process has multiple steps . In this post, we’ll kickstart the process by sharing some research topic ideas within the healthcare domain. This is the starting point, but to develop a well-defined research topic, you’ll need to identify a clear and convincing research gap , along with a well-justified plan of action to fill that gap.

If you’re new to the oftentimes perplexing world of research, or if this is your first time undertaking a formal academic research project, be sure to check out our free dissertation mini-course. In it, we cover the process of writing a dissertation or thesis from start to end. Be sure to also sign up for our free webinar that explores how to find a high-quality research topic.

Overview: Healthcare Research Topics

• Allopathic medicine
• Alternative /complementary medicine
• Veterinary medicine
• Physical therapy/ rehab
• Optometry and ophthalmology
• Pharmacy and pharmacology
• Public health
• Examples of healthcare-related dissertations

Allopathic (Conventional) Medicine

• The effectiveness of telemedicine in remote elderly patient care
• The impact of stress on the immune system of cancer patients
• The effects of a plant-based diet on chronic diseases such as diabetes
• The use of AI in early cancer diagnosis and treatment
• The role of the gut microbiome in mental health conditions such as depression and anxiety
• The efficacy of mindfulness meditation in reducing chronic pain: A systematic review
• The benefits and drawbacks of electronic health records in a developing country
• The effects of environmental pollution on breast milk quality
• The use of personalized medicine in treating genetic disorders
• The impact of social determinants of health on chronic diseases in Asia
• The role of high-intensity interval training in improving cardiovascular health
• The efficacy of using probiotics for gut health in pregnant women
• The impact of poor sleep on the treatment of chronic illnesses
• The role of inflammation in the development of chronic diseases such as lupus
• The effectiveness of physiotherapy in pain control post-surgery

Topics & Ideas: Alternative Medicine

• The benefits of herbal medicine in treating young asthma patients
• The use of acupuncture in treating infertility in women over 40 years of age
• The effectiveness of homoeopathy in treating mental health disorders: A systematic review
• The role of aromatherapy in reducing stress and anxiety post-surgery
• The impact of mindfulness meditation on reducing high blood pressure
• The use of chiropractic therapy in treating back pain of pregnant women
• The efficacy of traditional Chinese medicine such as Shun-Qi-Tong-Xie (SQTX) in treating digestive disorders in China
• The impact of yoga on physical and mental health in adolescents
• The benefits of hydrotherapy in treating musculoskeletal disorders such as tendinitis
• The role of Reiki in promoting healing and relaxation post birth
• The effectiveness of naturopathy in treating skin conditions such as eczema
• The use of deep tissue massage therapy in reducing chronic pain in amputees
• The impact of tai chi on the treatment of anxiety and depression
• The benefits of reflexology in treating stress, anxiety and chronic fatigue
• The role of acupuncture in the prophylactic management of headaches and migraines

Topics & Ideas: Dentistry

• The impact of sugar consumption on the oral health of infants
• The use of digital dentistry in improving patient care: A systematic review
• The efficacy of orthodontic treatments in correcting bite problems in adults
• The role of dental hygiene in preventing gum disease in patients with dental bridges
• The impact of smoking on oral health and tobacco cessation support from UK dentists
• The benefits of dental implants in restoring missing teeth in adolescents
• The use of lasers in dental procedures such as root canals
• The efficacy of root canal treatment using high-frequency electric pulses in saving infected teeth
• The role of fluoride in promoting remineralization and slowing down demineralization
• The impact of stress-induced reflux on oral health
• The benefits of dental crowns in restoring damaged teeth in elderly patients
• The use of sedation dentistry in managing dental anxiety in children
• The efficacy of teeth whitening treatments in improving dental aesthetics in patients with braces
• The role of orthodontic appliances in improving well-being
• The impact of periodontal disease on overall health and chronic illnesses

Tops & Ideas: Veterinary Medicine

• The impact of nutrition on broiler chicken production
• The role of vaccines in disease prevention in horses
• The importance of parasite control in animal health in piggeries
• The impact of animal behaviour on welfare in the dairy industry
• The effects of environmental pollution on the health of cattle
• The role of veterinary technology such as MRI in animal care
• The importance of pain management in post-surgery health outcomes
• The impact of genetics on animal health and disease in layer chickens
• The effectiveness of alternative therapies in veterinary medicine: A systematic review
• The role of veterinary medicine in public health: A case study of the COVID-19 pandemic
• The impact of climate change on animal health and infectious diseases in animals
• The importance of animal welfare in veterinary medicine and sustainable agriculture
• The effects of the human-animal bond on canine health
• The role of veterinary medicine in conservation efforts: A case study of Rhinoceros poaching in Africa
• The impact of veterinary research of new vaccines on animal health

Topics & Ideas: Physical Therapy/Rehab

• The efficacy of aquatic therapy in improving joint mobility and strength in polio patients
• The impact of telerehabilitation on patient outcomes in Germany
• The effect of kinesiotaping on reducing knee pain and improving function in individuals with chronic pain
• A comparison of manual therapy and yoga exercise therapy in the management of low back pain
• The use of wearable technology in physical rehabilitation and the impact on patient adherence to a rehabilitation plan
• The impact of mindfulness-based interventions in physical therapy in adolescents
• The effects of resistance training on individuals with Parkinson’s disease
• The role of hydrotherapy in the management of fibromyalgia
• The impact of cognitive-behavioural therapy in physical rehabilitation for individuals with chronic pain
• The use of virtual reality in physical rehabilitation of sports injuries
• The effects of electrical stimulation on muscle function and strength in athletes
• The role of physical therapy in the management of stroke recovery: A systematic review
• The impact of pilates on mental health in individuals with depression
• The use of thermal modalities in physical therapy and its effectiveness in reducing pain and inflammation
• The effect of strength training on balance and gait in elderly patients

Topics & Ideas: Optometry & Opthalmology

• The impact of screen time on the vision and ocular health of children under the age of 5
• The effects of blue light exposure from digital devices on ocular health
• The role of dietary interventions, such as the intake of whole grains, in the management of age-related macular degeneration
• The use of telemedicine in optometry and ophthalmology in the UK
• The impact of myopia control interventions on African American children’s vision
• The use of contact lenses in the management of dry eye syndrome: different treatment options
• The effects of visual rehabilitation in individuals with traumatic brain injury
• The role of low vision rehabilitation in individuals with age-related vision loss: challenges and solutions
• The impact of environmental air pollution on ocular health
• The effectiveness of orthokeratology in myopia control compared to contact lenses
• The role of dietary supplements, such as omega-3 fatty acids, in ocular health
• The effects of ultraviolet radiation exposure from tanning beds on ocular health
• The impact of computer vision syndrome on long-term visual function
• The use of novel diagnostic tools in optometry and ophthalmology in developing countries
• The effects of virtual reality on visual perception and ocular health: an examination of dry eye syndrome and neurologic symptoms

Topics & Ideas: Pharmacy & Pharmacology

• The impact of medication adherence on patient outcomes in cystic fibrosis
• The use of personalized medicine in the management of chronic diseases such as Alzheimer’s disease
• The effects of pharmacogenomics on drug response and toxicity in cancer patients
• The role of pharmacists in the management of chronic pain in primary care
• The impact of drug-drug interactions on patient mental health outcomes
• The use of telepharmacy in healthcare: Present status and future potential
• The effects of herbal and dietary supplements on drug efficacy and toxicity
• The role of pharmacists in the management of type 1 diabetes
• The impact of medication errors on patient outcomes and satisfaction
• The use of technology in medication management in the USA
• The effects of smoking on drug metabolism and pharmacokinetics: A case study of clozapine
• Leveraging the role of pharmacists in preventing and managing opioid use disorder
• The impact of the opioid epidemic on public health in a developing country
• The use of biosimilars in the management of the skin condition psoriasis
• The effects of the Affordable Care Act on medication utilization and patient outcomes in African Americans

Topics & Ideas: Public Health

• The impact of the built environment and urbanisation on physical activity and obesity
• The effects of food insecurity on health outcomes in Zimbabwe
• The role of community-based participatory research in addressing health disparities
• The impact of social determinants of health, such as racism, on population health
• The effects of heat waves on public health
• The role of telehealth in addressing healthcare access and equity in South America
• The impact of gun violence on public health in South Africa
• The effects of chlorofluorocarbons air pollution on respiratory health
• The role of public health interventions in reducing health disparities in the USA
• The impact of the United States Affordable Care Act on access to healthcare and health outcomes
• The effects of water insecurity on health outcomes in the Middle East
• The role of community health workers in addressing healthcare access and equity in low-income countries
• The impact of mass incarceration on public health and behavioural health of a community
• The effects of floods on public health and healthcare systems
• The role of social media in public health communication and behaviour change in adolescents

Examples: Healthcare Dissertation & Theses

While the ideas we’ve presented above are a decent starting point for finding a healthcare-related research topic, they are fairly generic and non-specific. So, it helps to look at actual dissertations and theses to see how this all comes together.

Below, we’ve included a selection of research projects from various healthcare-related degree programs to help refine your thinking. These are actual dissertations and theses, written as part of Master’s and PhD-level programs, so they can provide some useful insight as to what a research topic looks like in practice.

• Improving Follow-Up Care for Homeless Populations in North County San Diego (Sanchez, 2021)
• On the Incentives of Medicare’s Hospital Reimbursement and an Examination of Exchangeability (Elzinga, 2016)
• Managing the healthcare crisis: the career narratives of nurses (Krueger, 2021)
• Methods for preventing central line-associated bloodstream infection in pediatric haematology-oncology patients: A systematic literature review (Balkan, 2020)
• Farms in Healthcare: Enhancing Knowledge, Sharing, and Collaboration (Garramone, 2019)
• When machine learning meets healthcare: towards knowledge incorporation in multimodal healthcare analytics (Yuan, 2020)
• Integrated behavioural healthcare: The future of rural mental health (Fox, 2019)
• Healthcare service use patterns among autistic adults: A systematic review with narrative synthesis (Gilmore, 2021)
• Mindfulness-Based Interventions: Combatting Burnout and Compassionate Fatigue among Mental Health Caregivers (Lundquist, 2022)
• Transgender and gender-diverse people’s perceptions of gender-inclusive healthcare access and associated hope for the future (Wille, 2021)
• Efficient Neural Network Synthesis and Its Application in Smart Healthcare (Hassantabar, 2022)
• The Experience of Female Veterans and Health-Seeking Behaviors (Switzer, 2022)
• Machine learning applications towards risk prediction and cost forecasting in healthcare (Singh, 2022)
• Does Variation in the Nursing Home Inspection Process Explain Disparity in Regulatory Outcomes? (Fox, 2020)

Looking at these titles, you can probably pick up that the research topics here are quite specific and narrowly-focused , compared to the generic ones presented earlier. This is an important thing to keep in mind as you develop your own research topic. That is to say, to create a top-notch research topic, you must be precise and target a specific context with specific variables of interest . In other words, you need to identify a clear, well-justified research gap.

Need more help?

If you’re still feeling a bit unsure about how to find a research topic for your healthcare dissertation or thesis, check out Topic Kickstarter service below.

18 Comments

I need topics that will match the Msc program am running in healthcare research please

Hello Mabel,

I can help you with a good topic, kindly provide your email let’s have a good discussion on this.

Can you provide some research topics and ideas on Immunology?

Thank you to create new knowledge on research problem verse research topic

Help on problem statement on teen pregnancy

This post might be useful: https://gradcoach.com/research-problem-statement/

can you give me research titles that i can conduct as a school nurse

can you provide me with a research topic on healthcare related topics to a qqi level 5 student

Please can someone help me with research topics in public health ?

Hello I have requirement of Health related latest research issue/topics for my social media speeches. If possible pls share health issues , diagnosis, treatment.

I would like a topic thought around first-line support for Gender-Based Violence for survivors or one related to prevention of Gender-Based Violence

Please can I be helped with a master’s research topic in either chemical pathology or hematology or immunology? thanks

Can u please provide me with a research topic on occupational health and safety at the health sector

Good day kindly help provide me with Ph.D. Public health topics on Reproductive and Maternal Health, interventional studies on Health Education

may you assist me with a good easy healthcare administration study topic

May you assist me in finding a research topic on nutrition,physical activity and obesity. On the impact on children

I have been racking my brain for a while on what topic will be suitable for my PhD in health informatics. I want a qualitative topic as this is my strong area.

Hi, may I please be assisted with research topics in the medical laboratory sciences

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Top 15 journals in healthcare

What are your peers reading? We looked at usage data across key industry groups for 2023 to find our most-used journals among healthcare subscribers.

Journal of Parenteral And Enteral Nutrition

The  Journal of Parenteral and Enteral Nutrition  (JPEN) is the premier publication platform for the exchange of current research findings and dialogue of ideas in clinical nutrition and metabolic support.

JPEN is an official journal of the American Society for Parenteral and Enteral Nutrition (ASPEN). We publish original research, reviews, letters to the editor, and commentaries.

Seminars in Dialysis

Seminars in Dialysis  is a bimonthly nephrology journal focusing exclusively on cutting-edge clinical aspects of dialysis therapy for treatment of renal failure. Besides publishing papers by the most respected names in the field of dialysis, the journal has unique useful features, all designed to keep you current, including dialysis clinic, dialysis rounds, hemodialysis citations, editorials, opinions, and brief notes.

Hemodialysis International

Hemodialysis International  publishes quarterly issues containing original papers on clinical and experimental topics related to dialysis, in addition to the Annual Dialysis Conference Proceedings. This is a must-have journal for nephrologists, nurses and technicians worldwide.

Artificial Organs

Artificial Organs  is a transdisciplinary journal publishing the latest in research, development, translation, and clinical application of organ replacement/transplantation therapies, and organ machine perfusion preservation.

Our multi-disciplinary approach brings readers the full depth and breadth of the science and technology that continues to advance the Replacement, Recovery and Regeneration of tissues, organs, and organ systems.

Journal Der Deutschen Dermatologischen Gesellschaft

The  JDDG: Journal der Deutschen Dermatologischen Gesellschaft  is a dermatology journal publishing scientific papers from a wide range of disciplines, such as dermatovenereology, allergology, phlebology, dermatosurgery, dermatooncology, and dermatohistopathology. Also in JDDG: information on medical training, continuing education, a calendar of events, book reviews, and society announcements.

Papers can be submitted in German or English language. In the print version, all articles are published in German. In the online version, all key articles are published in English.

Nutrition in Clinical Practice

Nutrition in Clinical Practice  (NCP) publishes articles about the scientific basis and clinical application of nutrition and nutrition support. We aim to provide a practical focus that can enhance your daily work in delivering specialized nutrition support.

The journal contains comprehensive reviews, clinical research, case observations, and other types of papers written by experts in the field of nutrition and health care practitioners involved in the delivery of specialized nutrition support.

Transfusion

Transfusion  is the foremost publication in the world for new information regarding transfusion medicine. As the journal of the Association for the Advancement of Blood & Biotherapies, we report on the latest technical advances, discuss opposing viewpoints regarding controversial issues, and present key conference proceedings. In addition to blood banking and transfusion medicine topics, we present submissions concerning patient blood management, tissue transplantation and hematopoietic, cellular, and gene therapies.

Therapeutic Apheresis and Dialysis

As the official journal of the International Society for Apheresis and the Japanese Society for Apheresis,  Therapeutic Apheresis and Dialysis  covers research on apheresis and dialysis technologies and treatments.

We are committed to publishing content spanning from medicine, biology, and biophysics, to engineering, with a particular emphasis on how they converge in the development of new technologies and treatments resulting in improved patient outcomes.

Anaesthesia

Anaesthesia  is the official journal of the Association of Anaesthetists.

The journal is international in scope and comprehensive in coverage. We publish original articles on all aspects of general and regional anaesthesia, intensive care and pain therapy, including research on equipment.

Microsurgery

Microsurgery is an interdisciplinary journal for original contributions concerning surgery under microscopic magnification. We publish clinical studies, research papers, invited articles, relevant reviews, and other scholarly works from all related fields including orthopaedic surgery, otolaryngology, pediatric surgery, plastic surgery, urology, and vascular surgery.

Ophthalmic and Physiological Optics

Ophthalmic and Physiological Optics  (OPO) is a leading international, interdisciplinary journal that addresses basic and applied questions pertinent to contemporary research in vision science and optometry. OPO publishes original research papers, reviews, short communications and letters to the Editor and will interest researchers, educators and clinicians concerned with the development, use, and restoration of vision.

Cancer , an international interdisciplinary journal of the American Cancer Society, publishes high-impact, peer-reviewed original articles and solicited content on the latest clinical research findings. Spanning the breadth of oncology disciplines,  Cancer  delivers something for everyone involved in cancer research, risk reduction, treatment, and patient care.

Nephrology  is published 12 times per year by the Asian Pacific Society of Nephrology. It has a special emphasis on the needs of clinical nephrologists and those in developing countries. The journal publishes reviews and papers of international interest describing original research concerned with clinical and experimental aspects of nephrology.

British Journal of Haematology

An official journal of the British Society for Haematology, the  British Journal of Haematology  offers your research high visibility, receiving nearly 5 million article downloads annually. Content is delivered through a range of article types, including Original Papers, Reviews, BSH Guidelines, Images, and Wider Perspectives and Global View papers. The journal covers every specialism within haematology and welcomes high quality papers not just in clinical practice, but in the fields of basic science, translational research, and developments in haematology science.

Journal of Clinical Apheresis

The  Journal of Clinical Apheresis , the official publication of the American Society for Apheresis, publishes articles dealing with all aspects of hemapheresis and transfusion medicine. We welcome articles reporting on basic research and clinical applications of therapeutic plasma exchange, therapeutic cytapheresis, therapeutic absorption, blood component collection and transfusion, donor recruitment and safety, administration of hemapheresis centers, and innovative applications of hemapheresis technology. 

Is your company interested in subscribing to Wiley titles?  Learn more about our purchase models and access options.

10 Easy Steps to Tackling a Research Assignment

7 Key Themes at RAI 2020 Conference Anthropology and Geography: Dialogues Past, Present and Future

A Historical Look at Birth Control and Abortion from the Royal College of Physicians

A Kitchen Time Capsule: The Lady Sedley, and her recipe book, 1686

A Visual Journey Through the Archives: World Photography Day

A Wiley Network Spotlight on Historical Science this World Ocean Day

Before and After: Perspectives on the Digitization of Primary Sources

Celebrating International Archives Week 2020: Empowering Knowledge Societies

Celebrating the History of Environmental Science this Earth Day

Cochrane ‘Inside/Outside’: A 2022 Update from Carol Lefebvre

Related articles.

Help make research far less daunting with these 10 easy steps to tackling an assignment.

7 Key Themes at RAI 2020 Conference Anthropology and Geography: Dialogues Past, Present, and Future

Wellcome Library Research Fellow, Lesley A. Hall, discloses archives on birth control and abortion from the Royal College of Physicians.

“The Lady Sedley, her Receipt book, 1686” is just one of two million pages of documents belonging to the Royal College of Physicians that has been digitized by Wiley Digital Archives.

With over 400,000 photographs in Wiley Digital Archives, we invite you to journey with us through the archives as we celebrate World Photography Day, a day dedicated to Daguerre’s discovery.

Discover some key dates and interesting history about our exploration of Earth's oceans on World Ocean Day 2022

In this article, professors, researchers and our society partners share with us their perspectives on the value of digitizing archive collections.

Join us in celebrating International Archives Week 2020 as we promote the value of archives in research and empower societies.

Join Wiley in celebrating the history of environmental science this Earth Day (April 22). This year’s theme, brought to you by EarthDay.org, is Invest in Our Planet.

During the MLA 2022 Conference, Wiley sponsored its annual event featuring an update on Cochrane from Carol Lefevre, Independent Information Consultant.

Cochrane ‘Inside/Outside’: A 2022 Video Update from Carol Lefebvre

Coming Soon: New Wiley Digital Archive Focusing on Climate Change

Discover WDA: ENV - a new Wiley Digital Archive of historical data about climate change

FOR INDIVIDUALS

FOR INSTITUTIONS & BUSINESSES

WILEY NETWORK

ABOUT WILEY

Corporate Responsibility

Corporate Governance

Leadership Team

Cookie Preferences

Copyright @ 2000-2024  by John Wiley & Sons, Inc., or related companies. All rights reserved, including rights for text and data mining and training of artificial technologies or similar technologies.

Rights & Permissions

Privacy Policy

Terms of Use

• Open access
• Published: 22 August 2024

Factors influencing fidelity to guideline implementation strategies for improving pain care at cancer centres: a qualitative sub-study of the Stop Cancer PAIN Trial

• Tim Luckett 1 ,
• Jane Phillips 2 ,
• Meera Agar 1 , 3 ,
• Linda Richards 4 ,
• Najwa Reynolds 5 ,
• Maja Garcia 1 ,
• Patricia Davidson 6 ,
• Tim Shaw 7 ,
• David Currow 6 ,
• Frances Boyle 8 , 9 ,
• Lawrence Lam 10 ,
• Nikki McCaffrey 11 &
• Melanie Lovell 5 , 9  

BMC Health Services Research volume  24 , Article number:  969 ( 2024 ) Cite this article

77 Accesses

1 Altmetric

Metrics details

The Stop Cancer PAIN Trial was a phase III pragmatic stepped wedge cluster randomised controlled trial which compared effectiveness of screening and guidelines with or without implementation strategies for improving pain in adults with cancer attending six Australian outpatient comprehensive cancer centres ( n  = 688). A system for pain screening was introduced before observation of a ‘control’ phase. Implementation strategies introduced in the ‘intervention’ phase included: (1) audit of adherence to guideline recommendations, with feedback to clinical teams; (2) health professional education via an email-administered ‘spaced education’ module; and (3) a patient education booklet and self-management resource. Selection of strategies was informed by the Capability, Opportunity and Motivation Behaviour (COM-B) Model (Michie et al., 2011) and evidence for each strategy’s stand-alone effectiveness. A consultant physician at each centre supported the intervention as a ‘clinical champion’. However, fidelity to the intervention was limited, and the Trial did not demonstrate effectiveness. This paper reports a sub-study of the Trial which aimed to identify factors inhibiting or enabling fidelity to inform future guideline implementation initiatives.

The qualitative sub-study enabled in-depth exploration of factors from the perspectives of personnel at each centre. Clinical champions, clinicians and clinic receptionists were invited to participate in semi-structured interviews. Analysis used a framework method and a largely deductive approach based on the COM-B Model.

Twenty-four people participated, including 15 physicians, 8 nurses and 1 clinic receptionist. Coding against the COM-B Model identified ‘capability’ to be the most influential component, with ‘opportunity’ and ‘motivation’ playing largely subsidiary roles. Findings suggest that fidelity could have been improved by: considering the readiness for change of each clinical setting; better articulating the intervention’s value proposition; defining clinician roles and responsibilities, addressing perceptions that pain care falls beyond oncology clinicians’ scopes of practice; integrating the intervention within existing systems and processes; promoting patient-clinician partnerships; investing in clinical champions among senior nursing and junior medical personnel, supported by medical leaders; and planning for slow incremental change rather than rapid uptake.

Conclusions

Future guideline implementation interventions may require a ‘meta-implementation’ approach based on complex systems theory to successfully integrate multiple strategies.

Trial registration

Registry: Australian New Zealand Clinical Trials Registry; number: ACTRN 12615000064505; data: https://www.anzctr.org.au/Trial/Registration/TrialReview.aspxid=367236&isReview=true .

Peer Review reports

Pain is a common and burdensome symptom in people with cancer [ 1 ]. Barriers to pain care occur at all ‘levels’, including the patient and family (e.g., misconceptions regarding opioids), clinician (e.g. lack of expertise), service (e.g. inadequate referral processes) and healthcare system (e.g. lack of coordination) [ 2 , 3 , 4 , 5 , 6 , 7 , 8 ]. A recent systematic review suggests that around 40% of cancer patients with pain may not receive adequate management [ 9 ]. Research has demonstrated that routine screening and implementation of evidence-based guidelines has potential to improve quality of cancer pain care and outcomes [ 10 , 11 , 12 , 13 , 14 ]. However, experience suggests that clinicians are unlikely to utilise screening results or follow guidelines unless these are supported by targeted strategies [ 15 , 16 ].

The Stop Cancer PAIN Trial (ACTRN 12615000064505) was a phase III pragmatic stepped wedge cluster randomised controlled trial conducted between 2014 and 2019 which compared the effectiveness of screening and guidelines with or without implementation strategies for improving pain in adults with cancer attending six outpatient comprehensive cancer centres in Australia ( n  = 688) [ 17 , 18 ]. A pen/paper system to screen for pain using 0–10 numerical rating scales (NRS) for worst and average intensity over the past 24 h was introduced to each centre prior to observation of a ‘control’ phase, in which clinicians were also made aware of the Australian Cancer Pain Management in Adults guidelines [ 19 ]. At the beginning of the training phase, trial investigators presented at staff meetings on the importance of better managing pain and the rationale and evidence base for the intervention components. Implementation strategies (collectively termed the ‘intervention’) were then introduced in a ‘training’ phase and maintained during an ‘intervention’ phase as follows: (1) audit of adherence to key guideline recommendations [ 19 ] and feedback delivered to clinical teams in one or two cycles; (2) health professional education via a ‘Qstream’ email-administered ‘spaced education’ module [ 20 ]; and (3) a patient education booklet and self-management resource for completion together with a clinician that included goal setting, a pain diary and pain management plan [ 21 , 22 ]. Selection of these strategies was informed by the Capability, Opportunity and Motivation Behaviour (COM-B) Model of behaviour change [ 23 ], and evidence that each strategy had been separately effective for supporting guideline implementation for other health conditions. The intervention was supported at each centre by a consultant physician who agreed to be a ‘clinical champion’ [ 24 ].

As reported previously [ 18 ], the Stop Cancer PAIN Trial found no significant differences between the intervention and the control phases on the trial’s primary outcome - the proportion of patients with moderate-severe worst pain intensity who reported a 30% decrease at 1-week follow-up. Fidelity to the intervention was lower than anticipated and variable between centres: only 2/6 centres had two audit cycles rather than one; completion rates for the health professional spaced education varied from 12% to 74% between centres; and the proportion of patients reporting receipt of written information of any kind rose to an average of only 30% (20-44%) versus 22% (2-30%) in the control phase. Unexpectedly, secondary measures of mean, worst and average pain over a 4-week follow-up period improved by 0.5 standard deviation during control as well as intervention phases. However, the lack of a comparison group with no screening system made it difficult to conclude whether improvement in the control phase was due to effects from screening, a Hawthorne effect, or some other explanation.

The current paper reports a sub-study of the Stop Cancer PAIN Trial which aimed to identify factors influencing fidelity to the intervention that might warrant consideration by similar initiatives in the future.

The intervention, methods and results of the Stop Cancer PAIN trial have been described in previous open-access articles [ 17 , 18 ]. The sub-study used a qualitative approach with pragmatic orientation to enable in-depth exploration of factors influencing success from the perspectives of clinicians at each participating centre [ 25 ]. Clinician views canvassed at interview were considered the most efficient means of identifying barriers and enablers among complex contextual factors at each centre, including personnel’s knowledge, attitudes and beliefs towards pain care and the intervention.

The sub-study was approved by the Southwestern Sydney Local Health District Human Research Ethics Committee (HREC/14/LPOOL/479) as part of the overall trial. All participants gave written informed consent to participate.

Reporting adheres to the consolidated criteria for reporting qualitative research (COREQ) [ 26 ].

Participants

Participants were eligible if they were employed on a permanent basis either full- or part-time at a participating centre in a role that provided clinical care to cancer patients or patient-focused administrative support. The clinical champion at each centre was invited to participate by the research team. Other personnel were invited by means of email circulars and verbal invitations during meetings. Given the diverse range of roles at each centre, no limit was set on sample size to canvass as many perspectives as possible.

Data collection

Data were collected by means of semi-structured interviews conducted by one of two researchers, a female pharmacist with experience of medical education for pain management (LR), and a male social scientist with a doctorate (TL). Both interviewers had prior experience in qualitative research and knew some participants through their project roles.

Participants were fully aware of the study purpose before consenting. Interviews were conducted face-to-face or by telephone, with the participant and interviewer being the only people present. Interviews began with open questions about ‘what worked’ and ‘didn’t work’ across the intervention before focusing on each implementation strategy in more detail and important contextual factors at their centre (see Table  1 for a topic guide, which was developed specifically for this study). Interviewers explicitly invited criticism, expressing a tone of open enquiry and neutrality throughout. Prompts were used as necessary to explore factors identified by participants in more detail. Factors identified at previous interviews were raised at subsequent ones for verification, inviting participants to disagree or agree as they felt appropriate. No requests were received to return transcripts to participants for comment. Interviews were audio-recorded and transcribed verbatim.

Analysis used the framework method [ 27 ] and a largely deductive approach based on the same theoretical framework used during intervention design - the COM-B Model [ 23 ]. Based on a systematic review, the COM-B Model posits that behaviour change requires three conditions, namely ‘capability’ (including both psychological and physical capacity), ‘opportunity’ (all the factors that lie outside the individual that make the behaviour possible or prompt it) and ‘motivation’ (including habitual processes, emotional responding, as well as analytical decision-making). Initial line-by-line coding categorized data against these conditions according to which best described relationships between factors and behaviours within and across implementation strategies and the levels of patient, clinician and centre. While the COM-B model originally focused directly on human behaviour, it became clear during coding that behaviour was substantially influenced by centre, specialty and disciplinary factors, so these were also considered appropriate foci for coding against COM conditions. To enhance credibility, the same data were coded in different ways where multiple interpretations seemed plausible until coding of further interviews identified consistencies to help with disambiguation. Charting of codes for data within and between centres enabled mapping between the relative contributions made by each condition, summarised as lessons learned for guiding similar initiatives in the future. Dependability was increased by ensuring coding was conducted by two members of the research team (NR, MG) who had no previous involvement in the project but were experienced in qualitative research. Review and discussion with two team members who were involved in the project throughout (TL and ML) was intended to balance ‘outsider’ and ‘insider’ perspectives to guard against bias from preconceived interpretations whilst also referencing contextual understanding. Both Excel 2019 (Microsoft) and NVivo V12 (QSR) software were used to help manage different stages of the analytic process.

Twenty-four people participated across the six centres, ranging from one to six participants. Fifteen were physicians (of whom six were clinical champions), eight were nurses, and one was a clinic receptionist. This response rate ranged from 2 to 27% of eligible personnel at each centre. See Table  2 for a more detailed summary of participant roles at each centre. Interviews were a median of 20 min long, with an inter-quartile range of 13 to 28 min.

Capability, opportunity and motivation

Coding against the COM-B Model identified ‘capability’ to be the component having most influence over intervention success, with ‘opportunity’ and ‘motivation’ playing largely subsidiary roles.

Capabilities: Pertinent capabilities were reported to include: a pre-existing, centre-level culture of continuous improvement, communication pathways between senior management and other personnel, established roles and responsibilities for pain care among disciplines and specialties, systems and processes that could readily accommodate the intervention, and a culture of involving patients as partners in care. These capabilities influenced the degree to which personnel and patients had the opportunity and motivation to fully engage with the intervention.

Opportunity and motivation: These elements were most frequently discussed by participants in terms of ‘time’ that personnel could commit to pain care relative to other responsibilities. Clinical champions were perceived to play a critical role in supporting intervention success but were under-resourced at every centre and challenged by turnover in the role at two. In addition to more systemic drivers, individual personnel’s motivation was influenced by the degree to which they accepted the intervention’s value proposition at the outset and perceived this to be demonstrated over time.

Interactions between capability, opportunity and motivation are explored below in terms of their implications for similar future initiatives. Findings suggest that fidelity could have been improved by: considering the readiness for change of each clinical setting; better articulating the intervention’s value proposition; defining clinician roles and responsibilities, addressing perceptions that pain care falls beyond oncology clinicians’ scopes of practice; integrating the intervention within existing systems and processes; promoting patient-clinician partnerships; investing in clinical champions among senior nursing and junior medical personnel, supported by medical leaders; and planning for slow incremental change rather than rapid uptake.

Consider centres’ readiness for change

The degree to which centres had a pre-existing culture of continuous improvement was considered important in providing a fertile context for the intervention. At Centre 5, there was a consensus that change of any kind was difficult to instigate, even according to the head of department: “… because it’s new - because we’re so entrenched in our ways ” (C5P04 [Centre 5, participant 04] medical oncologist, head of department and clinical champion). At another, the complex centre-level nature of the intervention was perceived to pose particular challenges compared to oncology drug trials with which they were more familiar: “ we haven’t been a principal site [in a trial of this kind] previously and I think that’s sort of opened up some gaps in knowledge for us and some opportunities for learning in the future … what kind of support we’d need to come with that trial to help it be a success in this culture ” (C3P02 palliative care physician and clinical champion).

Articulate and deliver on the intervention’s value proposition

Interviews highlighted the importance of articulating the intervention’s value proposition to every member of the workforce and maintaining engagement by demonstrating benefits over time. At Centre 5, some participants perceived that the intervention had been imposed by management rather than generated from clinical priorities: “…senior staff say [to researchers] ‘come to our clinics, but we expect everyone else to do the work’ ” (C5P05 radiation oncologist). This was compounded by a perceived lack of communication about the project, which limited personnels’ opportunity to take a more active role even when they were motivated to do so: “ I would have facilitated [the intervention] … but I didn’t know about it ” (C5P01 nurse practitioner). Eliciting and maintaining engagement was said to be additionally challenged at this centre by high staff turnover, especially among junior medical officers on rotation: “ it was very accepted by the junior medical staff [but] I think, unfortunately, when there’s a relatively high turnover of staff … ” (C5P07 radiation oncology trainee). At two other centres, turnover among personnel required a transition in the role of clinical champion, interrupting support for the intervention while the new incumbents familiarised themselves with the role.

Across centres, participants reported reservations among some of their colleagues regarding the project’s fundamental premises, including the assumption that pain care needed improving at their centre (“ they actually felt this trial was a little bit insulting for their clinical skills. There was a bit of eye rolling and ‘of course we do that already!’ ” (C3P02 palliative care physician and clinical champion)) or that pain warranted a specific focus rather than symptoms more generally: “ I find it more useful when more than one symptom is targeted ” (C5P06 palliative care physician).

More specific criticism was also levelled at each of the intervention strategies as follows.

Pain screening

In the case of screening, two participants questioned the validity of a 0–10 numerical rating scale (NRS) for different reasons: “ sometimes getting the numbers breaks the flow of the narrative” (C6P04 medical oncologist); “they [patients] would say, ‘no, I’m not in pain but I have a lot of discomfort when I swallow’ - it was in the wording ” C5P02 registered nurse). Even one of the clinical champions felt that screening was redundant where pain was very severe: “ if someone is clearly in a pain crisis, you don’t need to be asking … you kind of know what number - they might tell you it’s 15 [out of 10] ” (C6P02 palliative care physician and clinical champion). Perceptions of the value of screening were also influenced by the degree to which it led to demonstrable improvements in pain care, which was undermined by problems with establishing an efficient process at some centres: “ I think I’ve still probably got stray [pain screening] forms on my desk ” (C3P06 palliative care physician). A lack of understanding among personnel and patients about how screening might lead to better pain outcomes was said to result in “ fatigue ” (C5P03 clinical nurse consultant [clinical nurse consultant]; C1P01 palliative care physician and clinical champion), manifest as a downward spiral of effort in, and value from, screening.

Audit and feedback

The audit and feedback strategy attracted limited attention from personnel at most centres: “ I don’t think that the audit and feedback were terribly noticeable ” (C4P01 medical oncologist and clinical champion). At the centre where only the palliative care department participated, one participant perceived baseline audit results to be acceptable and therefore demotivating for change: “[ the audit results showed] we were doing a good job even ahead of time … it did sort of make you think – ‘well where do we go from here?’ ” (C6P04 pain medicine physician). At another centre, motivation among personnel to improve on less favourable audit findings was perceived to depend on whether they prioritised pain care to start with: “ people have come up to me and said, ‘Gee, we really did very badly didn’t we?’ … but they’re not necessarily the people who don’t treat pain well - that’s the problem ” (C1P01 palliative care physician and clinical champion).

Spaced education for health professionals

Participants’ opinion on the value of the online spaced education depended on discipline and seniority, with nurses and junior medical officers reporting benefits “( it gave me a bit more confidence that I was on the right track” (C5P01 nurse practitioner)) but consultant physicians perceiving the knowledge level too “basic” (C6P04 pain medicine physician) or questioning advice from online spaced education that their responses were ‘wrong’: “…some of the multiple answers could have been equally valid” (C504 medical oncologist and clinical champion). Where consultants remained engaged, motivation was said to rely on cultivating “ competition” between colleagues (C602 palliative care physician and clinical champion). Inevitably, the voluntary nature of online spaced education also meant that only motivated personnel engaged to begin with.

Patient self-management resource

All participants who had used the patient self-management resource perceived at least some value. However, its use was limited by barriers relating to role and process considered below.

Define roles and responsibilities

Among the most commonly voiced barriers was a lack of clarity about which specialties and disciplines should be responsible for pain screening, patient education and management. This was usually described in terms of a ‘lack of time’ for pain care relative to other duties afforded greater priority within their scope of practice. Perspectives on roles and responsibilities are considered separately for each aspect of pain care as follows.

While most centres allocated the clinical task of pain screening to clinic receptionists, there was widespread reflection that this had been suboptimal. The only participating clinic receptionist felt that pain screening fell outside her area of responsibility: “but I’m an administrative person - I don’t have anything to do with pain management ” (C2P03 clinic receptionist). Clinician participants across disciplines similarly perceived that pain screening required clinical expertise to assist patients with reporting their pain and triage for urgent follow-up: “ you need somebody talking to the patients, rather than just handing the form, say ‘fill this in’ ” (C2P04 clinical nurse consultant). One centre that recognised this early on reallocated screening from an administrative to a nursing role, leading to substantial improvements in the completeness and quality of data: “ it made a big difference and certainly improved our ability to recognise people who had pain and allowed access for those people who were in severe pain to medications or at least an assessment … implementation through the clerical staff was not a long-term strategy ” (C1P01 palliative care physician and clinical champion).

Patient education

There was little consensus on which disciplines should be responsible for supporting patients to use the self-management resource, with medical personnel deferring to nurses and vice-versa. Role allocation was challenged by the diverse components within the resource, with each perceived to fall within a different scope of practice: “ pain is something I always do as an assessment … [but] … I’m not managing the pain … I’ll review and make recommendations and talk about the pain diaries and discussing their diary with their palliative care doctor or their general practitioner. And I would encourage that process. [But] I wouldn’t be the one that’s setting the goals on their daily activities and stuff ” (C5P01 nurse practitioner). Some oncology nursing roles were perceived to focus on chemo- or radiotherapy protocols to the exclusion of supportive care unless symptoms arose from, or impeded, treatment. Meanwhile, oncologists tended to interpret their role as solely focused on prescribing rather than also encompassing patient education: “ junior doctors only [have] 15 minutes to take a history and everything. [They] could enter in meds [into the patient resource] if everything else is done by someone else … part of me knows it’s [patient resource] important, but the other part of me - I just - when will I have time in my clinical practice to do it? ” (C5P05 radiation oncologist).

Pain management

Some oncologists viewed even pharmacological pain management as peripheral to their scope of practice when consultation time was short, prioritising cancer treatment instead. These participants viewed their role as limited to referring to palliative medicine or pain specialists, especially where pain was believed to have causes other than cancer: “ if the pain is a complex pain where the patient doesn’t have evidence of cancer, and it may be treatment-related, then in those scenarios we tend to divert to the chronic pain team ” (C5P07 radiation oncology advanced trainee). While participants from palliative care and pain medicine welcomed referrals for complex cases, they felt that oncologists sometimes referred for pain they could have easily managed themselves: “ what about some regular paracetamol? … These are things that you’d expect any junior doctors, never mind consultants [to have provided advice on] ” (C5P06 palliative care physician).

Integrate within existing systems and processes

Participants from several centres expressed a view that the intervention’s complex nature had proven overwhelming for systems and processes at their centres. At two centres, integration was especially challenged by broader infrastructure shifts and process failures that limited receptiveness to further changes. Participants at several centres emphasised the process-driven nature of oncology services and the challenge of changing established processes: “ they have got a pro forma that they use for chemo-immunotherapy review, and pain is not part of it, and that perhaps needs more of an organisational nuance … why doesn’t pain feature as a clinical outcome as part of the chemotherapy, immunotherapy review?” (C6P01 clinical nurse consultant). Participants emphasised the need to integrate pain care into existing processes to help personnel understand what was expected of them: “…nursing staff were getting them [screening forms] in the patient’s files and going, ‘what am I supposed to do with this?’ ” (C2P04 clinical nurse consultant). Moreover, centres’ focus on cancer treatment meant that pain care struggled to gain traction even when a process could be instituted: “ unless pain is the presenting complaint and is at the forefront it goes into those, sorts of, you know, the ‘other details’ ” (C5P06 palliative care physician). For the palliative care centre, where pain care was already prioritised, there were doubts about how the proposed process improved on those already in place: “ I generally ask pretty detailed questions about pain anyway [so don’t need patients to be screened in the waiting room] ” (C6P04 pain medicine physician).

Suggestions for better integrating the intervention included “in-building” (C3P04 medical oncologist) responsibility for the strategies within new staff roles or introducing the strategies gradually by means of a “ multistep process” (C5P04 medical oncologist, head of department and clinical champion). Features of two strategies were singled out as having positive potential for supporting existing processes of care. The patient resource was said to “ facilitate communication between the oncology teams and the palliative care team ” (C5P05 radiation oncologist) and serve as a “ visual cue ” (C3P02 medical oncologist) to cover educational topics that “ they might have otherwise forgotten ” (C2P01 palliative care physician and clinical champion). Participants also found the spaced education email administration, spacing and repetition “ easy to manage ” (C2P01 palliative care physician and clinical champion) within their daily routines.

Promote patient-clinician partnership on pain care

Several participants expressed surprise at the prevalence of moderate-severe pain in screening results, and acknowledged that this revealed under-reporting of pain in usual care. Under-reporting was perceived to stem partly from patient expectations that pain from cancer was “ normal ” (C4P03 nurse practitioner) and to be especially common in the context of certain generational or cultural attitudes towards pain and opioids (“ I certainly think there’s a cultural element but there’s also your elderly patients who you know have been through the war and they’re just used to coping with things and you just suck it up … it’s like a badge of honour to be able to say ‘I’m not one of these pill-takers ’” (C3P03 registered nurse [RN])) or when patients were concerned that reporting pain might reduce their fitness for anti-cancer treatment: “[ patients might think that] if I tell them honestly how crappy I am with other symptoms and pain and everything, then they might stop my chemo” (C3P02 palliative care physician). Several participants perceived that under-reporting was also due to patients taking an overly passive role in consultations: “[clinicians assume that] if the patient doesn’t bring it up, it’s not a problem for them and … then the patient [is] thinking ‘the doctor will only talk about important things that are important for me and I won’t mention it because obviously it’s not important’ ” (C3P02 palliative care physician and clinical champion).

The screening component of the intervention was considered to address under-reporting by “ normal[ising] ” pain care, thus encouraging disclosure. The patient resource was also considered helpful for building patient capability to partner with clinicians on pain management by “ encouraging self-efficacy ” (C2P01 palliative care physician and clinical champion) through the tools it provided and its positive message that “ you can get control of your pain ” (C3P02 palliative care physician and clinical champion). It was also perceived to help patients “ keep a record ” (C5P03 clinical nurse consultant) of breakthrough pain and analgesia to discuss in their consultation. However, some participants delineated patient groups who might be less able to use the resource, including those with lower educational levels who struggled to set goals and identify an ‘acceptable’ level of pain balanced against side-effects from pharmacological management. For these patients, it was suggested that too many resources could be overwhelming rather than supportive: “ it’s almost like, the more resources they have, the less resourced there are ” (C5P06 RN). At one centre with an especially diverse demographic, patients were said to require substantial support even to understand the purpose and process of pain screening: “ most [patients] look at you going ‘oh, do I have to do anything?’ … They don’t want to read the [instruction] page which is relatively simple ” (C2P03 clinic receptionist).

Invest in clinical champions

All participants perceived the role of clinical champion to be pivotal to the intervention’s success. Champions were perceived to have two major responsibilities: advocating for the intervention among colleagues to boost motivation and providing practical support to build capability.

To be effective advocates, champions were perceived to need support from senior management ( “[leadership of change] it’s got to happen from the top ” (C5P02 RN)) as well as established, cordial relationships with colleagues they could leverage to motivate engagement: “ it also relies on the champion’s personal relationship with the staff which you’re asking to perform these roles and trying to change their management ” (C1P01 palliative care physician and clinical champion). Where champions felt under-supported by management, they relied on moral support from the project team to sustain their advocacy work: “ being the champion, and sometimes being the nagging champion, it actually felt quite nice to have the back-up of other people ” (C1P01 palliative care physician and clinical champion). Both physicians and nurses perceived the champion role might better suit the scope of practice of a junior doctor or senior nurse rather than consultants, based on their willingness to engage and approachability: “ realistically, you’re probably always going to get more engagement with registrars compared to consultants, unless it’s their own trial ” (C5P07 radiation oncologist); “ just give it [the role] to the CNCs [clinical nurse consultants] because as a general rule they’re the best at everything and have the best relationships with the patient ” (C3P04 medical oncologist).

From a practical perspective, clinical champions were expected to provide human resources for establishing and supporting pain screening and patient education: “ you need a body ” (C2P04 clinical nurse consultant). Unfortunately, however, champions across centres reported having limited time protected for the role within their usual duties: “ there just wasn’t the manpower to do that here ” (C3P02 palliative care physician and clinical champion). One suggestion for boosting capacity was to narrow the focus to one clinic and delegate practical tasks to less senior delegates than required for advocacy to render the time commitment more cost-effective: “[ it] might have been better to focus on one clinic and have full-time … junior nurse ” (C5P05 radiation oncologist). This presented an opportunity to train more than one clinical champion to provide better coverage across shifts and safeguard against the risk of losing champions to staff turnover.

Increasing pain awareness is the first step: Plan for slow incremental change rather than rapid uptake

While the barriers above meant only modest practice changes could be achieved, champions at half the centres perceived incremental progress had been made through increasing awareness among personnel regarding pain care as a focus for improvement: “ I think just trying to make pain something that people think about was probably one of the better strategies ” (C1P01 palliative care physician and clinical champion); it’s more at the top of our minds to remember, to screen the pain at every visit ” (C2P01 palliative care physician and clinical champion); “ I think it has highlighted those issues for us and we now need to take this on ” (C5P04 medical oncologist, head of department and clinical champion). Both nursing and medical participants at Centre 5 emphasized the need to be persistent in striving for continuous improvement: “ I think to get practice change, even for well-motivated people, I think it just needs to be pushed … they’ve done similar things with hand washing for doctors and it’s finally getting through ” (C504 medical oncologist and clinical champion); “ it would take more than just one of these kind of programs to get people to change ” (C5P03 clinical nurse consultant). Encouragingly, participants at this and one other centre expected some clinicians to continue using the patient education booklet and resource after the project ended: “ I’d just love to continue using these booklets ” (C5P02 RN); “[the] patient-held resource has been useful and has been taken up by people and I think they will continue to use those ” (C6P02 palliative care physician and clinical champion).

This qualitative sub-study of a cluster randomized controlled trial identified centre-level capabilities to be the most influential factors impeding or facilitating guideline implementation strategies for improving pain care for outpatients with cancer. Findings suggest that future initiatives of this kind should: consider centre readiness for change; articulate and deliver on the intervention’s value proposition; define clinician roles and responsibilities; integrate the intervention within existing systems and processes; promote patient partnership; invest in the clinical champion role, drawing from senior nurses and junior doctors, with support from medical leaders and management; and design the initiative around slow incremental change rather than rapid uptake.

Our findings are largely consistent with those from an ethnographic study exploring factors influencing implementation of cancer pain guidelines in Korean hospital cancer units, which identified a ‘lack of receptivity for change’ to be a key barrier [ 28 ]. However, observations from the Korean study suggested that a lack of centre leadership and cultural norms regarding nursing hierarchy were the most important underlying factors, whereas our Australian sample focused more on constraints imposed by centre systems and processes and a lack of clarity regarding disciplinary roles. These factors were consistently emphasized regardless of participants’ discipline and seniority, including by one centre’s head of department. Consistent with these findings, a recent Australian qualitative sub-study of anxiety/depression guideline implementation in oncology centres found greater role flexibility to be a key factor underpinning organisational readiness for change [ 29 ]. This team also provided quantitative evidence consistent with our finding that centres’ readiness for change is associated with personnel’s perception of benefit from guideline implementation [ 30 ]. Future initiatives should work harder to persuade clinicians of the intervention’s rationale and evidence base prior to commencement, given that perceptions of coherence and effectiveness are key dimensions of acceptability required for clinicians to invest time and effort [ 31 ]. Since our Trial was conducted, evidence has emerged for an impact from cancer symptom screening on survival that could be used persuasively [ 32 ]. Furthermore, the spaced education module might be more acceptable if made adjustable to the knowledge levels of a broader range of clinicians.

Other studies on implementation of cancer pain guidelines [ 11 , 13 ] suggest that structured approaches to process change tend to be more successful than less prescriptive approaches of the kind taken in the Stop Cancer PAIN Trial. We provided centres with guideline implementation strategies but no clear guidance on how to integrate these within existing contexts - i.e. implementation of the implementation, or ‘meta-implementation’. It was wrongly assumed that clinical champions could support integration with centre processes based on their knowledge of local context, but this turned out to be unreasonable given champions’ limited time for the role and lack of training in change management. Like most research to date [ 33 , 34 ], our trial focused largely on the advocacy role played by clinical champions, neglecting more practical and time consuming aspects that our interviews identified to be just as important. We join others in calling for more research on the mechanisms by which clinical champions can optimally facilitate change and ways to maximize their efficacy through training and support [ 24 ]. This should include exploration of optimal models by which different aspects of the champion role might be shared between more than one person where no-one is available with all the necessary attributes, as well as ways to ensure sustainability after support from the project team is withdrawn.

Theory-based research suggests that adding complex interventions to complex healthcare systems creates dynamic interplay and feedback loops, making consequences hard to predict [ 35 ]. In the current trial, this was likely exacerbated by our attempt to combine multiple strategies targeting patient, clinician and centre levels. We chose each strategy based on evidence for its stand-alone efficacy, and combined strategies rather than used them singly with the intent of leveraging complementary mechanisms, as recommended by the COM-B Model and US Institute of Medicine [ 36 ]. However, findings from our interviews suggest that interactions between the strategies and local processes separated their spheres of influence, precluding intended synergies. The Stop Cancer PAIN Trial is not alone in having over-estimated the value of combining guideline implementation strategies; a recent systematic review found that 8 other multi-component interventions similarly demonstrated limited effects on guideline adherence and patient outcomes [ 37 ]. Collectively, these findings suggest that future attempts at combining strategies should consider complex systems theory as well as behaviour change frameworks at each of a number of stages [ 38 ]. Alternatively, a more manageable approach for most cancer centres might be to focus on just one component at a time, periodically reviewing progress against SMART goals and, depending on results, supplementing with additional components using plan-do-study cycles [ 39 ].

Given the challenges with integrating screening into centre processes, it seems unlikely that improvements in pain scores during the control phase reported in our primary results article were due to the spontaneous use of screening data in consultations [ 18 ]. Indeed, while routine use of patient-reported outcome measures (PROMs) in oncology has been researched for more than a quarter-century [ 40 ], benefits to patient outcomes have only recently been demonstrated in the context of electronically-administered PROMs (ePROMs) that enable remote self-reporting, real-time feedback to clinicians, and clinician-patient telecommunication [ 12 ]. Further research is needed on how best to support clinician engagement with ePROMs, including training on how to use results in partnership with patients to assist shared decision-making and self-management [ 41 ].

A worrying finding from the current study was that some or all aspects of pain care were perceived to fall between the scopes of practice for oncology clinicians from each discipline. Clinical practice guidelines emphasize the need for pain care to be inter-disciplinary in recognition of the need for comprehensive assessment, non-pharmacological as well as pharmacological management, and patient education and support for self-management [ 42 ]. While the patient self-management resource included in the intervention was perceived to support communication between clinicians and patients, its potential for assisting coordination of care between disciplines was limited where roles and responsibilities were not previously established. Our findings and other research suggest that future initiatives may benefit from ‘process mapping’ with clinicians to identify where clinical workflow and roles might be reconfigured to incorporate the various aspects of pain care in the most efficient ways that do not substantially add to workload [ 41 ].

Patient education has been proven to improve pain outcomes by clinical trials [ 43 , 44 ], and we have argued previously that supporting pain self-management should be core business for all clinicians working in cancer care [ 45 ]. The ‘coaching’ approach needed to empower patients to recognize themselves as ‘experts’ on their pain and equal partners with clinicians in its management is iterative rather than a single event, and is ideally built on established and ongoing therapeutic relationships of trust with a particular team member. However, findings from patient education research more generally suggest that patient education and behaviour change is also optimally supported when key messages are reinforced by differing disciplinary perspectives [ 46 ]. Results from the current study suggest that these principles of pain care need more formal recognition within the scope of practice of oncology clinicians to ensure they are afforded sufficient time alongside anti-cancer treatment and related supportive care. Findings also indicate that clinicians may require training in the person-centred, partnership-oriented aspects of pain care beyond the educational approach used in the Stop Cancer PAIN Trial and other research [ 47 ]. Such training should be repeated regularly to ensure it reaches the majority of personnel at cancer centres, allowing for turnover.

Limitations

The current study had several limitations. Transferability even within Australia is limited by a focus on metropolitan services in only three out of eight jurisdictions. Data relied on clinician perspectives, and the response rate was less than one quarter of personnel at each centre, with the disciplines and specialties of participants being unrepresentative of centre workforces. Over-sampling of medical compared to nursing personnel likely reflects the fact that all clinical champions were medical consultants, while the predominance of palliative care physicians among medical participants presumably arises from the central focus this specialty has on pain care. Notably, our sample included no perspectives from allied health disciplines, despite the important roles these can play in non-pharmacological pain management. Confirmability was threatened by the potential for cognitive bias among researchers towards a favourable view of the intervention given their long-standing investment as members of the project team. We attempted to offset this by explicitly inviting criticism of the intervention from participants, and having the initial analysis conducted by researchers with no prior involvement in the project. A final limitation concerns reliance on the COM-B Model for analysis rather than an alternative framework or more inductive approach. While the COM-B has been widely used to explore barriers and facilitators across a wide range of healthcare interventions, we applied the model in a somewhat novel way to systems and processes as well as individuals’ behaviour after finding that participants perceived their agency to be majorly constrained by these. An implementation framework such as the integrated-Promoting Action on Research Implementation in Health Service (i-PARIHS) framework (iPARIHS) [ 48 ] or Consolidated Framework for Implementation Research (CFIR) [ 49 ] would have conceived of factors and their relationships in alternative ways that might have proven equally informative [ 50 ].

This qualitative sub-study elucidated important factors influencing the success of guideline implementation strategies at six cancer centres in the Stop Cancer PAIN Trial. Findings underscore the value that a qualitative approach offers for understanding the role of context when evaluating complex interventions [ 51 ]. Ultimately, the Stop Cancer PAIN Trial may have been overly ambitious in the scale of its intervention, especially given limited resources available at each centre. Further research is needed to understand how multi-component guideline implementation strategies can be optimally introduced within the context of local roles, systems and processes.

Availability of data and materials

The qualitative interview datasets generated and analysed during the current study are not publicly available due to the conditions of ethical approval which acknowledge the risk of participant re-identification.

Van Den Beuken-Van MH, Hochstenbach LMJ, Joosten EAJ, Tjan-Heijnen VCG, Janssen DJA. Update on prevalence of pain in patients with cancer: systematic review and meta-analysis. J Pain Symptom Manag. 2016;51(6):1070–90.

Article   Google Scholar  

Luckett T, Davidson PM, Green A, Boyle F, Stubbs J, Lovell M. Assessment and management of adult cancer pain: a systematic review and synthesis of recent qualitative studies aimed at developing insights for managing barriers and optimizing facilitators within a comprehensive framework of patient care. J Pain Symptom Manag. 2013;46(2):229–53.

Oldenmenger WH, Sillevis Smitt PAE, van Dooren S, Stoter G, van der Rijt CCD. A systematic review on barriers hindering adequate cancer pain management and interventions to reduce them: a critical appraisal. Eur J Cancer. 2009;45(8):1370–80.

Article   PubMed   Google Scholar  

Jacobsen R, Sjogren P, Moldrup C, Christrup L. Physician-related barriers to cancer pain management with opioid analgesics: a systematic review. J Opioid Manag. 2007;3(4):207–14.

Jacobsen R, Moldrup C, Christrup L, Sjogren P. Patient-related barriers to cancer pain management: a systematic exploratory review. Scand J Caring Sci. 2009;23(1):190–208.

Jacobsen R, Liubarskiene Z, Moldrup C, Christrup L, Sjogren P, Samsanaviciene J. Barriers to cancer pain management: a review of empirical research. Medicina (Kaunas). 2009;45:427–33.

Fazeny B, Muhm M, Hauser I, Wenzel C, Mares P, Berzlanovich A, et al. Barriers in cancer pain management. Wiener Klinische Wochenschrift. 2000;112(22):978–81.

CAS   PubMed   Google Scholar  

Makhlouf SM, Pini S, Ahmed S, Bennett MI. Managing pain in people with cancer—a systematic review of the attitudes and knowledge of professionals, patients, caregivers and Public. J Cancer Educ. 2020;35(2):214–40.

Roberto A, Greco MT, Uggeri S, Cavuto S, Deandrea S, Corli O, et al. Living systematic review to assess the analgesic undertreatment in cancer patients. Pain Pract. 2022;22(4):487–96.

Brink-Huis A, van Achterberg T, Schoonhoven L, Brink-Huis A, van Achterberg T, Schoonhoven L. Pain management: a review of organisation models with integrated processes for the management of pain in adult cancer patients. J Clin Nurs. 2008;17(15):1986–2000.

Du Pen SL, Du Pen AR, Polissar N, Hansberry J, Kraybill BM, Stillman M, et al. Implementing guidelines for cancer pain management: results of a randomized controlled clinical trial. J Clin Oncol. 1999;17(1):361–70.

Article   CAS   PubMed   Google Scholar  

Govindaraj R, Agar M, Currow D, Luckett T. Assessing patient-reported outcomes in routine cancer clinical care using electronic administration and telehealth technologies: Realist synthesis of potential mechanisms for improving health outcomes. J Med Internet Res. 2023;25:e48483.

Article   PubMed   PubMed Central   Google Scholar  

Koesel N, Tocchi C, Burke L, Yap T, Harrison A. Symptom distress: implementation of palliative care guidelines to improve pain, fatigue, and anxiety in patients with advanced cancer. Clin J Oncol Nurs. 2019;23(2):149–55.

PubMed   Google Scholar  

Dulko D, Hertz E, Julien J, Beck S, Mooney K. Implementation of cancer pain guidelines by acute care nurse practitioners using an audit and feedback strategy. J Am Acad Nurse Pract. 2010;22(1):45–55.

Samuelly-Leichtag G, Adler T, Eisenberg E. Something must be wrong with the implementation of cancer-pain treatment guidelines. A lesson from referrals to a pain clinic. Rambam Maimonides Med J. 2019;10(3):18.

Lopez Lopez R, Camps Herrero C, Khosravi-Shahi P, Guillem Porta V, Carrato Mena A, Garcia-Foncillas J, et al. Oncologist’s knowledge and implementation of guidelines for breakthrough cancer pain in Spain: CONOCE study. Clin Transl Oncol. 2018;20(5):613–8.

Luckett T, Phillips J, Agar M, Lam L, Davidson PM, McCaffrey N, et al. Protocol for a phase III pragmatic stepped wedge cluster randomised controlled trial comparing the effectiveness and cost-effectiveness of screening and guidelines with, versus without, implementation strategies for improving pain in adults with cancer attending outpatient oncology and palliative care services: the Stop Cancer PAIN trial. BMC Health Serv Res. 2018;18(1):1–13.

Lovell MR, Phillips JL, Luckett T, Lam L, Boyle FM, Davidson PM, et al. Effect of cancer pain guideline implementation on pain outcomes among adult outpatients with cancer-related pain: a stepped wedge cluster randomized trial. JAMA Netw Open. 2022;5(2):1–12.

Lovell M, Luckett T, Boyle F, Stubbs J, Phillips J, Davidson PM, et al. Adaptation of international guidelines on assessment and management of cancer pain for the Australian context. Asia-Pac J Clin Oncol. 2015;11(2):170–7.

Phillips JL, Heneka N, Bhattarai P, Fraser C, Shaw T. Effectiveness of the spaced education pedagogy for clinicians’ continuing professional development: a systematic review. Med Educ. 2019;53(9):886–902.

Cancer Council Australia. Overcoming cancer pain: A guide for people with cancer, their families and friends 2013 [Available from: http://www.cancercouncil.com.au/wp-content/uploads/2014/01/Can487-Overcoming-Pain-NSW-Lores2.pdf .

Luckett T, Davidson PM, Green A, Marie N, Birch MR, Stubbs J, et al. Development of a cancer pain self-management resource to address patient, provider, and health system barriers to care. Palliat Support Care. 2019;17(4):472–8.

Michie S, Atkins L, West R. The behaviour change wheel: a guide to designing interventions. London: Silverback Publishing; 2014.

Google Scholar  

Santos WJ, Graham ID, Lalonde M, Demery Varin M, Squires JE. The effectiveness of champions in implementing innovations in health care: a systematic review. Implement Sci Commun. 2022;3(1):80.

Ramanadhan S, Revette AC, Lee RM, Aveling EL. Pragmatic approaches to analyzing qualitative data for implementation science: an introduction. Implement Sci Commun. 2021;2(1):70.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13(1):117.

Kim M, Jeong S, McMillan M, Higgins I. Translation of policy and guidelines into practice: lessons learned from implementation of the cancer pain management guideline. J Hosp Manage Health Policy. 2020;4.  https://doi.org/10.21037/jhmhp.2020.02.01 .

Geerligs L, Shepherd HL, Butow P, Shaw J, Masya L, Cuddy J, et al. What factors influence organisational readiness for change? Implementation of the Australian clinical pathway for the screening, assessment and management of anxiety and depression in adult cancer patients (ADAPT CP). Support Care Cancer. 2021;29(6):3235–44.

Faris MM, Shepherd HL, Butow PN, Kelly P, He S, Rankin N, et al. Staff- and service-level factors associated with organisational readiness to implement a clinical pathway for the identification, assessment, and management of anxiety and depression in adults with cancer. BMC Health Serv Res. 2023;23(1):866.

Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res. 2017;17(1):88.

Caminiti C, Maglietta G, Diodati F, Puntoni M, Marcomini B, Lazzarelli S, et al. The effects of patient-reported outcome screening on the survival of people with cancer: a systematic review and meta-analysis. Cancers. 2022;14(21):5470.

Morena AL, Gaias LM, Larkin C. Understanding the role of clinical champions and their impact on clinician behavior change: the need for causal pathway mechanisms. Front Health Serv. 2022;2:896885.

Miech EJ, Rattray NA, Flanagan ME, Damschroder L, Schmid AA, Damush TM. Inside help: an integrative review of champions in healthcare-related implementation. SAGE Open Med. 2018;6:2050312118773261.

Braithwaite J, Churruca K, Long JC, Ellis LA, Herkes J. When complexity science meets implementation science: a theoretical and empirical analysis of systems change. BMC Med. 2018;16:1–14.

Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Clinical Practice Guidelines We Can Trust. In: Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E, editors. Washington, D.C.: National Academies Press; 2011.

Bora A-M, Piechotta V, Kreuzberger N, Monsef I, Wender A, Follmann M, et al. The effectiveness of clinical guideline implementation strategies in oncology—a systematic review. BMC Health Serv Res. 2023;23(1):347.

McGill E, Er V, Penney T, Egan M, White M, Meier P, et al. Evaluation of public health interventions from a complex systems perspective: a research methods review. Soc Sci Med. 2021;272: 113697.

Institute for Healthcare Improvement. Quality improvement essentials toolkit. 2017.

Trowbridge RDW, Jay SJ, et al. Determining the effectiveness of a clinical-practice intervention in improving the control of pain in outpatient with cancer. Acad Med. 1997;72:798–800.

Howell D, Rosberger Z, Mayer C, Faria R, Hamel M, Snider A, et al. Personalized symptom management: a quality improvement collaborative for implementation of patient reported outcomes (PROs) in ‘real-world’ oncology multisite practices. J Patient Rep Outcomes. 2020;4(1):47.

Australian Adult Cancer Pain Management Guideline Working Party. Autralian clinical pathway for screening, assessment and management of cancer pain in adults Sydney, Australia: Cancer Council Australia. 2013. http://wiki.cancer.org.au/australia/Guidelines:Cancer_pain_management/Flowchart_overview .

Marie N, Luckett T, Davidson P, Lovell M, Lal S. Optimal patient education for cancer pain: a systematic review and theory-based meta-analysis. Support Care Cancer. 2013;21:3529–37.

Abernethy AP, Currow DC, Shelby-James T, Rowett D, May F, Samsa GP, et al. Delivery strategies to optimize resource utilization and performance status for patients with advanced life-limiting illness: results from the palliative care trial [ISRCTN 81117481]. J Pain Symptom Manage. 2013;45(3):488–505.

Lovell MR, Luckett T, Boyle FM, Phillips J, Agar M, Davidson PM. Patient education, coaching, and self-management for cancer pain. J Clin Oncol. 2014;32(16):1712–20.

Luckett T, Roberts MM, Smith T, Swami V, Cho JG, Wheatley JR. Patient perspectives on how to optimise benefits from a breathlessness service for people with COPD. NPJ Prim Care Respir Med. 2020;30(1). https://doi.org/10.1038/s41533-020-0172-4 .

Phillips JL, Heneka N, Lovell M, Lam L, Davidson P, Boyle F, et al. A phase III wait-listed randomised controlled trial of novel targeted inter-professional clinical education intervention to improve cancer patients’ reported pain outcomes (the Cancer Pain Assessment (CPAS) Trial): study protocol. Trials. 2019;20:1–12.

Harvey G, Kitson A. PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implement Sci. 2016;11(1):33.

Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated consolidated framework for implementation research based on user feedback. Implement Sci. 2022;17(1):1–16.

Geerligs L, Rankin NM, Shepherd HL, Butow P. Hospital-based interventions: a systematic review of staff-reported barriers and facilitators to implementation processes. Implement Sci. 2018;13(1):36.

Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of medical research council guidance. BMJ. 2021;374:n2061.

Download references

Acknowledgements

The authors would like to dedicate this article to the memory of Sally Fielding, who worked as a valued member of the project team throughout the Stop Cancer PAIN Trial. We would also like to acknowledge the contributions of project manager A/Prof Annmarie Hosie, data manager Dr Seong Cheah, and research assistant Layla Edwards.

This research was supported by a grant from the National Breast Cancer Foundation.

Author information

Authors and affiliations.

IMPACCT Centre—Improving Palliative, Aged and Chronic Care through Clinical Research and Translation, Faculty of Health, University of Technology Sydney (UTS), Building 10, 235 Jones St, Ultimo, Sydney, NSW, 2007, Australia

Tim Luckett, Meera Agar & Maja Garcia

School of Nursing and Centre for Healthcare Transformation, Queensland University of Technology (QUT), Brisbane, QLD, Australia

Jane Phillips

South West Sydney School of Clinical Medicine, University of New South Wales (UNSW), Sydney, NSW, Australia

The Limbic, Sydney, Australia

Linda Richards

Palliative Care Department, Greenwich Hospital, HammondCare, Sydney, NSW, Australia

Najwa Reynolds & Melanie Lovell

University of Wollongong, Wollongong, NSW, Australia

Patricia Davidson & David Currow

Charles Perkins Centre, School of Medical Sciences, The University of Sydney, Sydney, NSW, Australia

Patricia Ritchie Centre for Cancer Care and Research, The University of Sydney, Sydney, NSW, Australia

Frances Boyle

Northern Medical School, The University of Sydney, Sydney, NSW, Australia

Frances Boyle & Melanie Lovell

Macau University of Science and Technology, Macau, China

Lawrence Lam

Deakin Health Economics, Institute for Health Transformation, School of Health and Social Development, Deakin University, Melbourne, Australia

Nikki McCaffrey

You can also search for this author in PubMed   Google Scholar

Contributions

TL, JP, MA, PMD, TS, DCC, FB, LL, NM and ML contributed to the concept and design of this research. TL, LR, MR, MG and ML contributed to the acquisition, analysis or interpretation of the data. TL and ML contributed to drafting of the manuscript. All authors contributed to revisions of the manuscript and approved the final version.

Corresponding author

Correspondence to Tim Luckett .

Ethics declarations

Ethics approval and consent to participate.

This research was approved by the Southwestern Sydney Local Health District Human Research Ethics Committee (HREC/14/LPOOL/479). All participants gave written informed consent to participate.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Luckett, T., Phillips, J., Agar, M. et al. Factors influencing fidelity to guideline implementation strategies for improving pain care at cancer centres: a qualitative sub-study of the Stop Cancer PAIN Trial. BMC Health Serv Res 24 , 969 (2024). https://doi.org/10.1186/s12913-024-11243-1

Download citation

Received : 10 April 2024

Accepted : 25 June 2024

Published : 22 August 2024

DOI : https://doi.org/10.1186/s12913-024-11243-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation
• Qualitative

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

The role of data science in healthcare advancements: applications, benefits, and future prospects

Sri venkat gunturi subrahmanya.

1 Department of Electrical and Electronics Engineering, Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal, Karnataka India

Dasharathraj K. Shetty

2 Department of Humanities and Management, Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal, Karnataka India

Vathsala Patil

3 Department of Oral Medicine and Radiology, Manipal College of Dental Sciences, Manipal, Manipal Academy of Higher Education, Manipal Karnataka, India

B. M. Zeeshan Hameed

4 Department of Urology, Father Muller Medical College, Mangalore, Karnataka India

5 Department of Radiation Oncology, Massachusetts General Hospital, Boston, MA USA

Komal Smriti

Nithesh naik.

6 Department of Mechanical and Manufacturing Engineering, Manipal Institute of Technology, Manipal Academy of Higher Education, Manipal, Karnataka India

Bhaskar K. Somani

7 Department of Urology, University Hospital Southampton NHS Trust, Southampton, UK

Data science is an interdisciplinary field that extracts knowledge and insights from many structural and unstructured data, using scientific methods, data mining techniques, machine-learning algorithms, and big data. The healthcare industry generates large datasets of useful information on patient demography, treatment plans, results of medical examinations, insurance, etc. The data collected from the Internet of Things (IoT) devices attract the attention of data scientists. Data science provides aid to process, manage, analyze, and assimilate the large quantities of fragmented, structured, and unstructured data created by healthcare systems. This data requires effective management and analysis to acquire factual results. The process of data cleansing, data mining, data preparation, and data analysis used in healthcare applications is reviewed and discussed in the article. The article provides an insight into the status and prospects of big data analytics in healthcare, highlights the advantages, describes the frameworks and techniques used, briefs about the challenges faced currently, and discusses viable solutions. Data science and big data analytics can provide practical insights and aid in the decision-making of strategic decisions concerning the health system. It helps build a comprehensive view of patients, consumers, and clinicians. Data-driven decision-making opens up new possibilities to boost healthcare quality.

Introduction

The evolution in the digital era has led to the confluence of healthcare and technology resulting in the emergence of newer data-related applications [ 1 ]. Due to the voluminous amounts of clinical data generated from the health care sector like the Electronic Health Records (EHR) of patients, prescriptions, clinical reports, information about the purchase of medicines, medical insurance-related data, investigations, and laboratory reports, there lies an immense opportunity to analyze and study these using recent technologies [ 2 ]. The huge volume of data can be pooled together and analyzed effectively using machine-learning algorithms. Analyzing the details and understanding the patterns in the data can help in better decision-making resulting in a better quality of patient care. It can aid to understand the trends to improvise the outcome of medical care, life expectancy, early detection, and identification of disease at an initial stage and required treatment at an affordable cost [ 3 ]. Health Information Exchange (HIE) can be implemented which will help in extracting clinical information across various distinct repositories and merge it into a single person’s health record allowing all care providers to access it securely. Hence, the organizations associated with healthcare must attempt to procure all the available tools and infrastructure to make use of the big data, which can augment the revenue and profits and can establish better healthcare networks, and stand apart to reap significant benefits [ 4 , 5 ]. Data mining techniques can create a shift from conventional medical databases to a knowledge-rich, evidence-based healthcare environment in the coming decade.

Big data and its utility in healthcare and medical sciences have become more critical with the dawn of the social media era (platforms such as Facebook and Twitter) and smartphone apps that can monitor personal health parameters using sensors and analyzers [ 6 , 7 ]. The role of data mining is to improvise the stored user information to provide superior treatment and care. This review article provides an insight into the advantages and methodologies of big data usage in health care systems. It highlights the voluminous data generated in these systems, their qualities, possible security-related problems, data handling, and how this analytics support gaining significant insight into these data set.

Search strategy

A non-systematic review of all data science, big data in healthcare-related English language literature published in the last decade (2010–2020) was conducted in November 2020 using MEDLINE, Scopus, EMBASE, and Google Scholar. Our search strategy involved creating a search string based on a combination of keywords. They were: “Big Data,” “Big Data Analytics,” “Healthcare,” “Artificial Intelligence,” “AI,” “Machine learning,” “ML,” “ANN,” “Convolutional Networks,” “Electronic Health Records,” “EHR,” “EMR,” “Bioinformatics,” and “Data Science.” We included original articles published in English.

Inclusion criteria

• Articles on big data analytics, data science, and AI.
• Full-text original articles on all aspects of application of data science in medical sciences.

Exclusion criteria

• Commentaries, reviews, and articles with no full-text context and book chapters.
• Animal, laboratory, or cadaveric studies.

The literature review was performed as per the above-mentioned strategy. The evaluation of titles and abstracts, screening, and the full article text was conducted for the chosen articles that satisfied the inclusion criteria. Furthermore, the authors manually reviewed the selected article’s references list to screen for any additional work of interest. The authors resolved the disagreements about eligibility for a consensus decision after discussion.

Knowing more about “big data”

Big data consists of vast volumes of data, which cannot be managed using conventional technologies. Although there are many ways to define big data, we can consider the one defined by Douglas Laney [ 8 ] that represents three dimensions, namely, volume, velocity, and variety (3 Vs). The “big” in big data implies its large volume. Velocity demonstrates the speed or rate at which data is processed. Variety focuses on the various forms of structured and raw data obtained by any method or device, such as transaction-level data, videos, audios, texts, emails, and logs. The 3 Vs became the default description of big data, while many other Vs are added to the definition [ 9 ]. “Veracity” remains the most agreed 4th “V.” Data veracity focuses on the accuracy and reliability of a dataset. It helps to filter through what is important and what is not. The data with high veracity has many records that are valuable to analyze and that contribute in a meaningful way to the overall results. This aspect poses the biggest challenge when it comes to big data. With so much data available, ensuring that it is relevant and of high quality is important. Over recent years, big data has become increasingly popular across all parts of the globe.

Big data needs technologically sophisticated applications that use high-end computing resources and Artificial Intelligence (AI)-based algorithms to understand such huge volumes of data. Machine learning (ML) approaches for automatic decision-making by applying fuzzy logic and neural networks will be added advantage. Innovative and efficient strategies for dealing with data, smart cloud-based applications, effective storage, and user-friendly visualization are required for big data to gain practical insights [ 10 ].

Medical care as a repository for big data

Healthcare is a multilayered system developed specifically for preventing, diagnosing, and treating diseases. The key elements of medical care are health practitioners (physicians and nurses), healthcare facilities (which include clinics, drug delivery centers, and other testing or treatment technologies), and a funding agency that funds the former. Health care practitioners belong to different fields of health such as dentistry, pharmacy, medicine, nursing, psychology, allied health sciences, and many more. Depending on the severity of the cases, health care is provided at many levels. In all these stages, health practitioners need different forms of information such as the medical history of the patient (data related to medication and prescriptions), clinical data (such as data from laboratory assessments), and other personal or private medical data. The usual practice for a clinic, hospital, or patient to retain these medical documents would be maintaining either written notes or in the form of printed reports [ 11 ].

The clinical case records preserve the incidence and outcome of disease in a person’s body as a tale in the family, and the doctor plays an integral role in this tale [ 12 ]. With the emergence of electronic systems and their capacity, digitizing medical exams, health records, and investigations is a common procedure today. In 2003, the Institute of Medicine, a division in the National Academies of Sciences and Engineering coined the term “Electronic Health Records” for representing an electronic portal that saves the records of the patients. Electronic health records (EHRs) are automated medical records of patients related to an individual’s physical/mental health or significant reports that are saved in an electronic system and used to record, send, receive, store, retrieve, and connect the medical personnel and patient with medical services [ 13 ].

Open-source big data platforms

It is an inefficient idea to work with big data or vast volumes of data into storage considering even the most powerful computers. Hence, the only logical approach to process large quantities of big data available in a complex form is by spreading and processing it on several parallel connected nodes. Nevertheless, the volume of the data is typically so high that a large number of computing machines are needed in a reasonable period to distribute and finish processing. Working with thousands of nodes involves coping with issues related to paralleling the computation, spreading of data, and manage failures. Table ​ Table1 1 shows the few open sources of big data platforms and their utilities for data scientists.

source big data platforms and their utilities

Data mining

Data types can be classified based on their nature, source, and data collection methods [ 14 ]. Data mining techniques include data grouping, data clustering, data correlation, and mining of sequential patterns, regression, and data storage. There are several sources to obtain healthcare-related data (Fig.  1 ). The most commonly used type (77%) is the data generated by humans (HG data) which includes Electronic Medical Records (EMR), Electronic Health Records (EHR), and Electronic Patient Records (EPR). Online data through Web Service (WS) is considered as the second largest form of data (11%) due to the increase in the number of people using social media day by day and current digital development in the medical sector [ 15 ]. Recent advances in the Natural Language Processing (NLP)-based methodologies are also making WS simpler to use [ 16 ]. The other data forms such as Sensor Data (SD), Big Transactional Data (BTD), and Biometric Data (BM) make around 12% of overall data use, but wearable personal health monitoring devices’ prominence and market growth [ 17 ] may need SD and BM data.

Sources of big data in healthcare

Applications of analytics in healthcare

There are six areas of applications of analytics in healthcare (Fig.  2 ) including disease surveillance, health care management and administration, privacy protection and fraud detection, mental health, public health, and pharmacovigilance. Researchers have implemented data extraction for data deposition and cloud-based computing, optimizing quality, lowering costs, leveraging resources, handling patients, and other fields.

Various applications of data science in healthcare

Disease surveillance

It involves the perception of the disease, understanding its condition, etiology (the manner of causation of a disease), and prevention (Fig.  3 ).

The disease analysis system

Information obtained with the help of EHRs, and the Internet has a huge prospect for disease analysis. The various surveillance methods would aid the planning of services, evaluation of treatments, priority setting, and the development of health policy and practice.

Image processing of healthcare data from the big data point of view

Image processing on healthcare data offers valuable knowledge about anatomy and organ functioning and identifies the disease and patient health conditions. The technique currently has been used for organ delineation, identification of lung tumors, diagnosis of spinal deformity, detection of arterial stenosis, detection of an aneurysm, etc. [ 18 ]. The wavelets technique is commonly used for image processing techniques such as segmentation, enhancement, and noise reduction. The use of artificial intelligence in image processing will enhance aspects of health care including screening, diagnosis, and prognosis, and integrating medical images with other types of data and genomic data will increase accuracy and facilitate early diagnosis of diseases [ 18 , 19 ]. The exponential increase in the count of medical facilities and patients has led to better use of clinical settings of computer-based healthcare diagnostics and decision-making systems.

Data from wearable technology

Multi-National Companies like Apple and Google are working on health-based apps and wearable technology as part of a broader range of electronic sensors, the so-called IoT, and toolkits for healthcare-related apps. The possibility of collecting accurate medical data on real-time (e.g., mood, diet followed, exercise, and sleep cycles patterns), linked to physiological indicators (e.g., heart rate, calories burned, level of blood glucose, cortisol levels), is perhaps discrete and omnipresent at minimum cost, unrelated to traditional health care. “True Colors” is a wearable designed to collect continuous patient-centric data with the accessibility and acceptability needed to allow for accurate longitudinal follow-up. More importantly, this system is presently being piloted as a daily health-monitoring substitute.

Medical signal analytics

Telemetry and the devices for the monitoring of physiological parameters generate large amounts of data. The data generated generally are retained for a shorter duration, and thus, extensive research into produced data is neglected. However, advancements in data science in the field of healthcare attempt to ensure better management of data and provide enhanced patient care [ 20 – 23 ].

The use of continuous waveform in health records containing information generated through the application of statistical disciplines (e.g., statistical, quantitative, contextual, cognitive, predictive, etc.) can drive comprehensive care decision-making. Data acquisition apart from an ingestion-streaming platform is needed that can control a set of waveforms at various fidelity rates. The integration of this waveform data with the EHR’s static data results in an important component for giving analytics engine situational as well as contextual awareness. Enhancing the data collected by analytics will not just make the method more reliable, but will also help in balancing predictive analytics’ sensitivity and specificity. The signal processing species must mainly rely on the kind of disease population under observation.

Various signal-processing techniques can be used to derive a large number of target properties that are later consumed to provide actionable insight by a pre-trained machine-learning model. Such observations may be analytical, prescriptive, or predictive. Such insights can be furthermore built to activate other techniques such as alarms and physician notifications. Maintaining these continuous waveforms–based data along with specific data obtained from the remaining sources in perfect harmony to find the appropriate patient information to improve diagnosis and treatments of the next generation can be a daunting task [ 24 ]. Several technological criteria and specifications at the framework, analytical, and clinical levels need to be planned and implemented for the bedside implementation of these systems into medical setups.

Healthcare administration

Knowledge obtained from big data analysis gives healthcare providers insights not available otherwise (Fig.  4 ). Researchers have implemented data mining techniques to data warehousing as well as cloud computing, increasing quality, minimizing costs, handling patients, and several other fields of healthcare.

Role of big data in accelerating the treatment process

Data storage and cloud computing

Data warehousing and cloud storage are primarily used for storing the increasing amount of electronic patient-centric data [ 25 , 26 ] safely and cost-effectively to enhance medical outcomes. Besides medical purposes, data storage is utilized for purposes of research, training, education, and quality control. Users can also extract files from a repository containing the radiology results by using keywords following the predefined patient privacy policy.

Cost and quality of healthcare and utilization of resources

The migration of imaging reports to electronic medical recording systems offers tremendous potential for advancing research and practice on radiology through the continuous updating, incorporation, and exchange of a large volume of data. However, the heterogeneity in how these data can be formatted still poses major challenges. The overall objective of NLP is that the natural human language is translated into structured with a standardized set of value choices that are easily manipulated into subsections or searches for the presence or absence of a finding through software, among other things [ 27 ].

Greaves et al. [ 28 ] analyzed sentiment (computationally dividing them into categories such as optimistic, pessimistic, and neutral) based on the online response of patients stating their overall experience to predict healthcare quality. They found an agreement above 80% between online platform sentiment analysis and conventional paper-based quality prediction surveys (e.g., cleanliness, positive conduct, recommendation). The newer solution can be a cost-effective alternative to conventional healthcare surveys and studies. The physician’s overuse of screening and testing often leads to surplus data and excess costs [ 29 ]. The present practice in pathology is restricted by the emphasis on illness. Zhuang et al. [ 29 ] compared the disease-based approach in conjunction with database reasoning and used the data mining technique to build a decision support system based on evidence to minimize the unnecessary testing to reduce the total expense of patient care.

Patient data management

Patient data management involves effective scheduling and the delivery of patient care during the period of a patient’s stay in a hospital. The framework of patient-centric healthcare is shown in Fig.  5 . Daggy et al. [ 30 ] conducted a study on “no shows” or missing appointments that lead to the clinical capability that has been underused. A logistical regression model is developed using electronic medical records to estimate the probabilities of patients to no-show and show the use of estimates for creating clinical schedules that optimize clinical capacity use while retaining limited waiting times and clinical extra-time. The 400-day clinical call-in process was simulated, and two timetables were developed per day: the conventional method, which assigns one patient per appointment slot, and the proposed method, which schedules patients to balance patient waiting time, additional time, and income according to no-show likelihood.

Elemental structure of patient-centric healthcare and ecosystem

If patient no-show models are mixed with advanced programming approaches, more patients can be seen a day thus enhancing clinical performance. The advantages of implementation of planning software, including certain methodologies, should be considered by clinics as regards no-show costs [ 30 ].

A study conducted by Cubillas et al. [ 31 ] pointed out that it takes less time for patients who came for administrative purposes than for patients for health reasons. They also developed a statistical design for estimating the number of administrative visits. With a time saving of 21.73% (660,538 min), their model enhanced the scheduling system. Unlike administrative data/target finding patients, a few come very regularly for their medical treatment and cover a significant amount of medical workload. Koskela et al. [ 32 ] used both supervised and unsupervised learning strategies to identify and cluster records; the supervised strategy performed well in one cluster with 86% accuracy in distinguishing fare documents from the incorrect ones, whereas the unsupervised technique failed. This approach can be applied to the semi-automate EMR entry system [ 32 ].

Privacy of medical data and fraudulency detection

The anonymization of patient data, maintaining the privacy of the medical data and fraudulency detection in healthcare, is crucial. This demands efforts from data scientists to protect the big data from hackers. Mohammed et al. [ 33 ] introduced a unique anonymization algorithm that works for both distributed and centralized anonymization and discussed the problems of privacy security. For maintaining data usefulness without the loss of any data privacy, the researchers further proposed a model that performed far better than the traditional K-anonymization model. In addition to this, their algorithm could also deal with voluminous, multi-dimensional datasets.

A mobile-based cloud-computing framework [ 34 ] of big data has been introduced to overcome the shortcomings of today’s medical records systems. EHR data systems are constrained due to a lack of interoperability, size of data, and privacy. This unique cloud-based system proposed to store EHR data from multiple healthcare providers within the facility of an internet provider to provide authorized restricted access to healthcare providers and patients. They used algorithms for encryption, One Time Password (OTP), or a 2-factor authentication to ensure data security.

The analytics of the big data can be performed using Google’s efficient tools such as big query tools and MapReduce. This approach will reduce costs, improve efficiency, and provide data protection compared to conventional techniques that are used for anonymization. The conventional approach generally leaves data open to re-identification. Li et al. in a case study showed that hacking can make a connection between tiny chunks of information as well as recognize patients [ 35 ]. Fraud detection and abuse (i.e., suspicious care behavior, deliberate act of falsely representing facts, and unwanted repeated visits) make excellent use of big data analytics [ 36 ].

By using data from gynecology-based reports, Yang et al. framed a system that manually distinguishes characteristics of suspicious specimens from a set of medical care plans that any doctor would mostly adopt [ 37 ]. The technique was implemented on the data from Taiwan’s Bureau of National Health Insurance (BNHI), where the proposed technique managed to detect 69% of the total cases as fraudulent, enhancing the current model, which detected only 63% of fraudulent cases. To sum up, the protection of patient data and the detection of fraud are of significant concern due to the growing usage of social media technology and the propensity of people to place personal information on these platforms. The already existing strategies for anonymizing the data may become less successful if they are not implemented because a significant section of the personal details of everyone is now accessible through these platforms.

Mental health

According to National Survey conducted on Drug Use and Health (NSDUH), 52.2% of the total population in the United States (U.S.) was affected by either mental problems or drug addiction/abuse [ 38 ]. In addition, approximately 30 million suffer from panic attacks and anxiety disorders [ 39 ].

Panagiotakopoulos et al. [ 40 ] developed a data analysis–focused treatment technique to help doctors in managing patients with anxiety disorders. The authors used static information that includes personal information such as the age of the individual, sex, body and skin types, and family details and dynamic information like the context of stress, climate, and symptoms to construct static and dynamic information based on user models. For the first three services, relationships between different complex parameters were established, and the remaining one was mainly used to predict stress rates under various scenarios. This model was verified with the help of data collected from twenty-seven volunteers who are selected via the anxiety assessment survey. The applications of data analytics in the disease diagnosis, examination, or treatment of patients with mental wellbeing are very different from using analytics to anticipate cancer or diabetes. In this case, the data context (static, dynamic, or non-observable environment) seems to be more important compared to data volume [ 39 ].

The leading cause of perinatal morbidity and death is premature birth, but an exact mechanism is still unclear. The research carried by Chen et al. [ 41 ] intended to investigate the risk factors of preterm use of neural networks and decision tree C5.0 data mining. The original medical data was obtained by a specialist study group at the National University of Taiwan from a prospective pregnancy cohort. A total of 910 mother–child dyads from 14,551 in the original data have been recruited using the nest case–control design. In this data, thousands of variables are studied, including basic features, medical background, the climate and parents’ occupational factors, and the variables related to children. The findings suggest that the main risk factors for pre-born birth are multiple births, blood pressure during pregnancy, age, disease, prior preterm history, body weight and height of pregnant women, and paternal life risks associated with drinking and smoking. The results of the study are therefore helpful in the attempt to diagnose high-risk pregnant women and to provide intervention early to minimize and avoid early births in parents, healthcare workers, and public health workers [ 41 , 42 ].

Public health

Data analytics have also been applied to the detection of disease during outbreaks. Kostkova et al. [ 43 ] analyzed online records based on behavior patterns and media reporting the factors that affect the public as well as professional patterns of search-related disease outbreaks. They found distinct factors affecting the public health agencies’ skilled and layperson search patterns with indications for targeted communications during emergencies and outbreaks. Rathore et al. [ 44 ] have suggested an emergency tackling response unit using IoT-based wireless network of wearable devices called body area networks (BANs). The device consists of “intelligent construction,” a model that helps in processing and decision making from the data obtained from the sensors. The system was able to process millions of users’ wireless BAN data to provide an emergency response in real-time.

Consultation online is becoming increasingly common and a possible solution to the scarcity of healthcare resources and inefficient delivery of resources. Numerous online consultation sites do however struggle to attract customers who are prepared to pay and maintain them, and health care providers on the site have the additional challenge to stand out from a large number of doctors who can provide similar services [ 45 ]. In this research, Jiang et al. [ 45 ] used ML approaches to mine massive service data, in order (1) to define the important characteristics related to patient payment rather than free trial appointments, (2) explore the relative importance of those features, and (3) understand how these attributes work concerning payment, whether linearly or not. The dataset refers to the largest online medical consultation platform in China, covering 1,582,564 consultation documents among patient pairs between 2009 and 2018. The results showed that compared with features relating to reputation as a physician, service-related features such as quality of service (e.g., intensity of consultation dialogue and response rate), the source of patients (e.g., online vs offline patients), and the involvement of patients (e.g., social returns and previous treatments revealed). To facilitate payment, it is important to promote multiple timely responses in patient-provider interactions.

Pharmacovigilance

Pharmacovigilance requires tracking and identification of adverse drug reactions (ADRs) after launch, to guarantee patient safety. ADR events’ approximate social cost per year reaches a billion dollars, showing it as a significant aspect of the medical care system [ 46 ]. Data mining findings from adverse event reports (AERs) revealed that mild to lethal reactions might be caused in paclitaxel among which docetaxel is linked with the lethal reaction while the remaining 4 drugs were not associated with hypersensitivity [ 47 ] while testing ADR’s “hypersensitivity” to six anticancer agents [ 47 ]. Harpaz et al. [ 46 ] disagreed with the theory that adverse events might be caused not just due to a single medication but also due to a mixture of synthetic drugs. It is found that there is a correlation between a minimum of one drug and two AEs or two drugs and one AE in 84% of AERs studies. Harpaz R et al. [ 47 ] improved precision in the identification of ADRs by jointly considering several data sources. When using EHRs that are available publicly in conjunction with the AER studies of the FDA, they achieved a 31% (on average) increase in detection [ 45 ]. The authors identified dose-dependent ADRs with the help of models built from structured as well as unstructured EHR data [ 48 ]. Of the top 5 ADR-related drugs, 4 were observed to be dose-related [ 49 ]. The use of text data that is unstructured in EHRs [ 50 ]; pharmacovigilance operation was also given priority.

ADRs are uncommon in conventional pharmacovigilance, though it is possible to get false signals while finding a connection between a drug and any potential ADRs. These false alarms can be avoided because there is already a list of potential ADRs that can be of great help in potential pharmacovigilance activities [ 18 ].

Overcoming the language barrier

Having electronic health records shared worldwide can be beneficial in analyzing and comparing disease incidence and treatments in different countries. However, every country would use their language for data recording. This language barrier can be dealt with the help of multilingual language models, which would allow diversified opportunities for Data Science proliferation and to develop a model for personalization of services. These models will be able to understand the semantics — the grammatical structure and rules of the language along with the context — the general understanding of words in different contexts.

For example: “I’ll meet you at the river bank.”

“I have to deposit some money in my bank account.”

The word bank means different things in the two contexts, and a well-trained language model should be able to differentiate between these two. Cross-lingual language model trains on multiple languages simultaneously. Some of the cross lingual language models include:

mBERT — the multilingual BERT which was developed by Google Research team.

XLM — cross lingual model developed by Facebook AI, which is an improvisation over mBERT.

Multifit — a QRNN-based model developed by Fast.Ai that addresses challenges faced by low resource language models.

Millions of data points are accessible for EHR-based phenotyping involving a large number of clinical elements inside the EHRs. Like sequence data, handling and controlling the complete data of millions of individuals would also become a major challenge [ 51 ]. The key challenges faced include:

• The data collected was mostly either unorganized or inaccurate, thus posing a problem to gain insights into it.
• The correct balance between preserving patient-centric information and ensuring the quality and accessibility of this data is difficult to decide.
• Data standardization, maintaining privacy, efficient storage, and transfers require a lot of manpower to constantly monitor and make sure that the needs are met.
• Integrating genomic data into medical studies is critical due to the absence of standards for producing next-generation sequencing (NGS) data, handling bioinformatics, data deposition, and supporting medical decision-making [ 52 ].
• Language barrier when dealing data

Future directions

Healthcare services are constantly on the lookout for better options for improving the quality of treatment. It has embraced technological innovations intending to develop for a better future. Big data is a revolution in the world of health care. The attitude of patients, doctors, and healthcare providers to care delivery has only just begun to transform. The discussed use of big data is just the iceberg edge. With the proliferation of data science and the advent of various data-driven applications, the health sector remains a leading provider of data-driven solutions to a better life and tailored services to its customers. Data scientists can gain meaningful insights into improving the productivity of pharmaceutical and medical services through their broad range of data on the healthcare sector including financial, clinical, R&D, administration, and operational details.

Larger patient datasets can be obtained from medical care organizations that include data from surveillance, laboratory, genomics, imaging, and electronic healthcare records. This data requires proper management and analysis to derive meaningful information. Long-term visions for self-management, improved patient care, and treatment can be realized by utilizing big data. Data Science can bring in instant predictive analytics that can be used to obtain insights into a variety of disease processes and deliver patient-centric treatment. It will help to improvise the ability of researchers in the field of science, epidemiological studies, personalized medicine, etc. Predictive accuracy, however, is highly dependent on efficient data integration obtained from different sources to enable it to be generalized. Modern health organizations can revolutionize medical therapy and personalized medicine by integrating biomedical and health data. Data science can effectively handle, evaluate, and interpret big data by creating new paths in comprehensive medical care.

OOpen access funding provided by Manipal Academy of Higher Education, Manipal.

Declarations

The authors declare no competing interests.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.