systematic review in nursing education

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Open access
• Published: 28 September 2023

Effectiveness of virtual reality in nursing education: a systematic review and meta-analysis

• Kai Liu 1 ,
• Weiwei Zhang 2 ,
• Ting Wang 1 &
• Yanxue Zheng 1  

BMC Medical Education volume  23 , Article number:  710 ( 2023 ) Cite this article

11k Accesses

15 Citations

11 Altmetric

Metrics details

This study aims to assess the transformative potential of Virtual Reality (VR) has shown significant potential in transforming nursing education by providing immersive and interactive learning experiences. Our objective is to systematically evaluate and conduct a meta-analysizes on the impact effect of virtual reality technology in teaching nursing students.

To achieve this, we conducted comprehensive computer searches on platforms including of PubMed, Web of Science, Wiley Online Library, Zhiwang database, Wanfang database, and China Biomedical Literature Service (SinoMed), were conducted to collect randomized controlled trial studies on the use of virtual reality’s technology for teaching nursing students built up to until March 2023., and the Cochrane Furthermore, the quality of the included literature was assessed evaluated using the quality evaluation criteria specified for randomized controlled trial studies within the Cochrane provided in the evaluation handbook manual. In addition, a meta-analysis was performed using Review Manager 5.3 software.

The aggregate outcomes from a total of 12 randomized controlled trials, encompassing including 1167 students, indicate were included. Meta-analysis results showed that virtual reality technology significantly enhances could better improve nursing students’’ theoretical knowledge [(SMD = 0.97, 95% CI [0.48, 1.46], p < 0.001)], practical skills (SMD = 0.52, 95% CI [0.33, 1.46], p < 0.001), skill retention, (SMD = 0.52, 95% CI [0.33, 0.71], p < 0.001), and satisfaction levels (SMD = 1.14, 95% CI [0.85, 1.43], p < 0.001), in comparison with traditional or alternative teaching methodologies. However, no statistically significant impact was observed on the enhancement of critical thinking skills (SMD = 0.79, 95% CI [-0.05, 1.64], p = 0.07) among nursing students.

Our findings underscore that compared to conventional teaching methods, virtual reality offers superior potential in advancing nursing students’ theoretical knowledge, practice proficiencies, and overall satisfaction, while not yielding a significant advantage in enhancing critical thinking skills. The incorporated literature consisted exclusively of randomized controlled trials, albeit a subset of these studies omitted descriptions of the allocation concealment strategy.

Peer Review reports

Introduction

Virtual reality (VR) technology, stemming from mathematical reasoning and scientific experimentation, assumes a pivotal role as a universal and strategic tool for comprehending, reshaping, and innovating the tangible world [ 1 , 2 ]. VR involves creating a computer-generated simulation of a three-dimensional image or environment that individuals can interact with as if it were real or physical. Thus, this interaction is achieved through specialized electronic equipment, such as a helmet with an integrated screen or gloves equipped with sensors. This holds within the evolving landscape of education reform and heightened aspirations for elevated higher education standards. Significantly accentuating this trajectory, VR’s educational potential is thrust into the spotlight [ 3 ], propelling it into a burgeoning realm of investigation with expansive practical applications. Notably in the domain of nursing education, this technology garners increasing attention from medical education scholars who seek to curtail teaching expenses and mitigate instructional hazards while upholding pedagogical excellence [ 4 , 5 , 6 ].

Amid the rapid evolution of information technology, novel advancements like VR are ushering in a fresh pedagogical approach to nursing education [ 7 ]. This technique engenders an immersive, interactive encounter for nursing students, replicating authentic clinical scenarios and furnishing them with robust hands-on training experiences, all while circumventing direct patient engagement. This not only economizes the valuable time of clinical nursing professionals but also mitigates the predicaments associated with conventional patient interaction in pedagogy [ 8 ], thereby addressing the dearth of clinical educational resources. Nursing students interact with VR through immersive experiences that simulate real-life clinical scenarios. Using specialized VR equipment, such as VR headsets or gloves with sensors, nursing students can engage with virtual patients, medical equipment, and healthcare environments.

This allows them to practice clinical skills, make decisions in simulated patient care situations, and explore various medical scenarios. VR technology enables nursing students to actively participate in realistic learning experiences, enhancing their understanding of theoretical concepts and providing hands-on training in a safe and controlled environment. Besides, the interactive dimension inherent to VR technology augments the didactic process, bestowing nursing students with an intuitively enriched learning experience [ 9 ]. Through its immersive interactivity, VR effectively eradicates the dullness of conventional teaching, and rigid teaching methodologies, kindling greater scholarly enthusiasm among nursing students and markedly improving their operational skills [ 10 ]. By proficiently role-modeling clinical environments, VR fosters a profound sense of engagement, inherently nurturing nursing students’ professional self-conception [ 11 ], strengthening their sense of commitment and calling, and galvanizing autonomous knowledge exploration. Moreover, the interactive operability at the core of VR fundamentally amplifies the intrinsic impetus for nursing students to actively pursue learning [ 12 ].

Jang et al. [ 13 ] demonstrated that the interactive and operational attributes of VR outperform 3D videos in enhancing students’ comprehension and assimilation of knowledge. Within the realm of nursing education, there exists a significant challenge in devising a curriculum that encompasses both depth and breadth, rectifying the limitations intrinsic to conventional pedagogy. This challenge seeks to seamlessly guide nursing students in the transition from fragmented textual knowledge to its clinical application, concurrently conserving invaluable clinical teaching resources and nurturing adept nursing practitioners [ 14 ].

Virtual reality (VR) is the use of computer technology to create an interactive three-dimensional (3D) world, which gives users a sense of spatial presence. In nursing education, VR has been used to help optimize teaching and learning processes. In the contemporary landscape of nursing education reform, this stands as a crucial and intricate juncture demanding resolution. Consequently, a symbiotic fusion of nursing education with the ever-evolving era is indispensable for nurturing adaptable nursing professionals capable of propelling the progress and expansion of both nursing education and clinical practice. As the trajectory of nursing education evolves, the integration of VR instruction emerges as a definitive trend [ 15 ], yet it remains in its nascent exploratory phase, necessitating substantial engagement from nursing educators to chart a scientifically sound path toward effective VR nursing education.

Hence, this study meticulously reviewed the existing literature concerning the integration of VR into nursing education curricula, with a focus on both current and prospective nursing educators. Employing a systematic evaluation and subsequent meta-analysis, we aimed to discern optimal approaches in nursing student instruction, by identifying essential attributes of nursing education and furnishing a guiding trajectory for future research about virtual reality-infused nursing educational programs.

Study design

This study employed “The Preferred Reporting Items for Systematic Reviews and Meta-Analyses”, otherwise known as the principles of the PRISMA statement in the reporting of the meta-analysis [ 16 ].

Search strategy

We conducted computer-based searches across multiple databases including PubMed, Web of Science, Wiley Online Library, Zhiwang database, Wanfang database, and China Biomedical Literature Service (SinoMed). The search spanned from the establishment of each database up to March 2023, employing a search strategy that combined both free terms and subject descriptors. Equally, by employing literature tracing techniques, we located pertinent studies. On top of that, the search strategy encompassed a fusion of free terms and subject descriptors. Within the English databases, search terms include “virtual reality/patient simulat*/virtual patient*/virtual simulation” and “education, nursing /nurs*education/education of nursing.” As an illustrative instance, the search strategy utilized for PubMed is outlined in Fig.  1 .

Inclusion and exclusion criteria

Inclusion criteria.

Adhering to the population, intervention, control, outcomes, and study design (PICOS) principles, we methodically screened the literature based on the following criteria: (1) study population (P): nursing students; (2) intervention (I): experimental groups in the studies that employed VR technology for instruction, implementing the four essential components delineated by Sherman and Craig (virtual world, immersion, sensory feedback, interactivity) to establish varying degrees of immersion through diverse VR platforms [ 17 ]; (3) control measures (C): control groups were exposed to conventional pedagogical techniques (comprising classroom lectures, demonstrations, model-based instruction, etc.) or non-VR simulation scenarios (encompassing high-fidelity/low-fidelity simulation, mannequin simulation, etc.); (4) outcomes (O): evaluated outcome indicators, which includes theoretical knowledge scores, practical skills scores, satisfaction levels, and critical thinking abilities; (5) study design (S): exclusively encompassing randomized controlled trials.

Exclusion criteria

We excluded the following types of literature during the selection process: (1) conference papers, abstracts, and catalogs; (2) duplicate publications; (3) literature containing errors, incomplete study data, or being inaccessible for comprehensive analysis; (4) literature with inaccessible full text; (5) literature not published in English or Chinese; (6) Other types of literature besides experimental studies.

Study selection and data extraction

Two researchers conducted an independent screening of the literature based on the inclusion and exclusion criteria. Initially, they reviewed the titles and abstracts of the literature, excluding those that did not meet the inclusion criteria. Subsequently, a meticulous assessment of the full texts was undertaken for the remaining literature, leading to their inclusion upon alignment with the set criteria. Ultimately, the literature that adhered to the inclusion criteria was definitively identified. In instances of discrepancies, a third researcher was consulted to facilitate the decision-making process, and any information gaps were addressed by reaching out to the original authors whenever feasible. When confronted with duplicate publications, precedence was given to the Chinese literature.

Following the identification of the literature, two researchers autonomously undertook the task of data extraction. They meticulously extracted data from the selected studies in alignment with the prescribed data extraction protocols as outlined in the Cochrane Handbook for the Evaluation of Intervention Systems [ 18 ]. Furthermore, the extracted information encompassed a range of elements including authors, year of publication, country, study population, number of participants, and number of interventionists.

Correspondingly, we adhered to the risk of bias assessment of the principles for Randomized Controlled Trials (RCT) as prescribed by the Cochrane Collaboration Network Handbook on Systematic Evaluation of Interventional Studies, version 5.1. Two investigators autonomously conducted evaluations of the included RCTs, while any disparities were resolved through deliberation or consultation with a third investigator when necessary.

Study quality and risk of bias assessment

Two investigators independently conducted assessments in accordance with the evaluation criteria outlined in Cochrane Evaluation Manual 5.1.0 [ 19 ]. In instances where discrepancies arose, a third investigator was consulted to facilitate resolution through discussion or consultation. The assessment encompassed the following items: ① generation of randomized sequences; ② allocation scheme concealment; ③ blinding of investigators and subjects; ④ blinding of assessors; ⑤ completeness of data; ⑥ selective reporting; and ⑦ other biases. Thus, the classification of risk levels for each element was divided into high risk, low risk, and unclear. This study, therefore, elucidates the rationale behind each judgment by the assessment framework of the Review Manager 5.3 program.

Statistical methods

We conducted the meta-analysis of the included literature utilizing Review Manager 5.3 software and Endnote software. Continuous variables were elucidated through mean difference (MD), standardized mean difference (SMD), and a 95% confidence interval (CI). Statistical significance and differences between the experimental and control groups were established at P < 0.05. Besides, we assessed the heterogeneity of the meta-analysis results utilizing the I 2 quantitative test. A value of I 2  < 50% indicated low heterogeneity, enabling the selection of a fixed-effect model. Conversely, an I 2 value ≥ 50% denoted high heterogeneity, warranting the adoption of a random-effect model. For I 2 values exceeding 75%, significant heterogeneity within the meta-analysis results was ascertained.

Characteristics of the study population (study selection)

A computer search of PubMed, Web of Science, Wiley Online Library, Zhiwang database, Wanfang database and China Biomedical Literature Service (SinoMed) retrieved 2179 papers; titles and abstracts were reviewed and 152 articles were selected considering the inclusion and exclusion criteria. After excluding 140 papers that did not meet the inclusion criteria, 12 papers were finally selected [ 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 ]. As shown in Fig.  2 .

Literature screening process and result

Characteristics of research on educational interventions using virtual reality for current and future health personnel (research characteristics)

This study includes general characteristics of 12 studies of educational interventions using virtual reality; detailed information is provided in Table  1 In terms of study design characteristics, all 12 studies were randomized controlled studies. The number of study participants was 585 for the experimental group and 582 for the control group, for a total of 1167 participants.

Methodological quality assessment of intervention studies

The risk of bias assessment is summarized in Fig.  3 below. 9 of the 12 studies described detailed information related to randomization, and 7 studies lacked clarity regarding allocation concealment. The detailed risk of bias assessment is shown in Fig.  4 .

Overall risk of bias analysis of included studies

Risk of bias analysis of each included study

Meta-analysis results

Theoretical knowledge.

Eleven studies [20–25,27−31] evaluated the effectiveness of VR technology in theoretical knowledge levels. The results showed a high heterogeneity of the included studies (p < 0.001, I2 = 92%), so a random effects model was used. The combined results showed that the use of VR technology was effective in improving students’ theoretical knowledge compared to other traditional nursing teaching methods (SMD = 0.97, 95% CI [0.48, 1.46], P < 0.001, Fig.  5 ) .

Impact of VR technology on nursing students’ theoretical knowledge

Practical skills

Four studies [ 24 , 28 , 29 , 31 ]evaluated the effectiveness of VR technology in practice skills. The results showed that there was no heterogeneity in the included studies (P = 0.34, I2 = 10%), so a fixed-effects model was used. The combined results showed a statistically significant difference compared to other traditional nursing teaching methods (SMD = 0.15, 95% CI [-0.21, 0.51], P < 0.001, Fig.  6 ).

Impact of VR technology on nursing students’ practice skills

Satisfaction

Four studies [ 20 , 21 , 23 , 25 ]evaluated the effect of VR technology on nursing teaching satisfaction, and the results showed that there was no heterogeneity in the included studies (P = 0.36, I 2  = 7%), so a fixed-effects model was used. The combined results showed a significant difference in the improvement of satisfaction with nursing education with VR technology compared to the control teaching modality (SMD = 1.14, 95% CI [0.85, 1.43], P < 0.001, Fig.  7 ).

Impact of VR technology on academic satisfaction of nursing students

Critical thinking

Three studies [ 26 , 30 , 31 ]evaluated the effect of VR technology on the application of critical thinking among nursing students, but the results showed high heterogeneity in the included studies (P < 0.001, I 2  = 93%), so a random effects model was used. The combined results showed no significant difference in the improvement of satisfaction in nursing education with VR technology compared to the control teaching modality (SMD = 0.79, 95% CI [-0.05, 0.46], p = 0.07, Fig.  8 ).

Impact of VR technology on nursing students’ critical thinking

The effect of the application of VR technology on nursing students

The results of the aforementioned meta-analysis revealed that the utilization of VR technology in nursing student education yielded more effective enhancement in theoretical knowledge in comparison to other conventional teaching methods, and this distinction achieved statistical significance (P < 0.05). Contemporaneously, the theoretical component of nursing education stands as a crucial underpinning for nurses to translate their knowledge into clinical competence, constituting an integral part of nursing instruction. In a bid to bolster the teaching effectiveness of the theoretical course, nursing educators have incorporated virtual reality technology. Similarly, West Chavez et al. [ 32 ] demonstrated how VR technology increased student engagement by immersing them in realistic learning experiences that closely mirrored their real environment.

Moreover, a qualitative exploration into VR technology’s integration within nursing education [ 33 ] suggests that amalgamating VR technology with conventional nursing pedagogy enables students to interact with objects within a specific virtual teaching environment, engendering equivalent sentiments and experiences as in the real environment. This immersion empowers learners to fathom what they have imbibed and how to practically apply that knowledge. Furthermore, aligned with Kolb’s experiential learning model [ 34 ], nursing students glean insights from their virtual world experiences akin to real-life occurrences, thereby garnering more immediate and enduring outcomes. This explains the observed escalation in theoretical knowledge. According to Pei Ning Woon et al. [ 35 ] ,Virtual reality may be a viable teaching strategy to improve knowledge acquisition, but it is presently suitable for supplementing conventional teaching methods. However, the incorporation of VR technology into nursing instruction not only captivates nursing students’ dedication to learning but also fortifies their knowledge and skills, serving as a foundational requisite for propelling nursing students’ transition from knowledge-based to competence-driven paradigms.

VR technology effect on the practical skills of nursing students

In comparison with conventional or alternative nursing education methodologies, the application of VR technology in teaching exhibits a disparity in nursing practical skills (P > 0.05). That said, virtual simulation technology distinctly elucidates operational intricacies, facilitating accelerated and comprehensive knowledge assimilation among students, thereby augmenting instructional efficacy. A pertinent example stems from Hong Kong Polytechnic University’s successful integration of virtual reality technology in nasogastric tube placement training, attesting to the technology’s safety, flexibility, and interactivity advantages that accelerates the learning curve for this procedure [ 36 ].

Consistent with prior studies, Jefferson (2022) devised a high-fidelity simulation (HFS) course, whereby participants exemplified heightened learning retention and enhanced practical aptitude levels [ 37 ]. Consequently, future endeavors should prioritize refining the technical effectiveness of VR teaching environments, as well as enhancing the transference of acquired practical skills from virtual to clinical settings. In addition, empowering nursing students to engage with patients within virtual environments considerably enhances their clinical technique comprehension and enables them to navigate clinical scenarios during practice. Congruently, certain techniques unfeasible for real patients can be practiced on virtual patients, thereby streamlining the transformation process from theory to practice, student to practitioner, and classroom to clinical settings. This redresses inherent limitations within traditional practical training approaches, subsequently boosting nursing students’ learning efficiency [ 38 ].

VR technology’s improvement effect on nursing students’ academic satisfaction

VR technology imbues learning with vividness and imagery, exuding a potent contagious influence that substantially heightens nursing students’ perceptual acuity, enthusiasm, and active engagement in learning, effectively positioning them at the center of the learning process. When applied to nursing instruction, VR amplifies the liveliness of educational content, fostering augmented interest in independent learning among nursing students, and catalyzing a shift from passivity to activeness in their journey of intellectual voyage. Apart from that, this technology propels students beyond the mere exercise of operational skills within virtual contexts; rather, it serves as a platform where they refine not only operational proficiencies but also their clinical reasoning, decision-making acumen, and aptitude for troubleshooting practical clinical problems [ 39 ]. At the same time, VR offers a robust milieu for collaborative learning, promoting teacher-student interactions. Through group discussions and cooperative endeavors facilitated by VR, educators, and learners deepen their problem analysis and critical thinking, honing their cognitive capacities, and inadvertently enhancing students’ prowess in collaborative communication. This progressive enhancement of collaborative communication inadvertently bolsters the academic contentment of nursing students [ 40 ].

The effect of VR technology on the level of nursing students’ critical thinking

The findings of this study revealed a lack of statistically significant difference (P > 0.05) in the improvement of critical thinking with in nursing education through the implementation of VR technology. This outcome is chiefly attributed to the paucity of research on the relationship between virtual learning environments and the cultivation of critical thinking skills among college students. Also, the obscurity surrounding the determinants influencing college students’ critical thinking development within virtual learning environments, the intricate processes underlying each determinant’s impact, and the varying degrees of influence exerted by these determinants, remain unresolved.

By the same token, capitalizing on the rapid advancements in virtual reality technology, researchers have turned their attention towards harnessing virtual learning environments to foster and nurture college students’ critical thinking prowess. Notably, Kandi et al. (2020) conducted an experimental inquiry that unveiled that architecture students immersed in a virtual learning environment exhibited enhancements in their design, review, and innovation skills. Furthermore, the study ascertained that a virtual reality game design simulator empowered students to pinpoint design errors more effectively and subsequently excel in critical thinking tests [ 41 ]. In like manner, Kang (2020) developed a virtual reality nursing course and found that it facilitated students’ critical thinking development and independent learning skills [ 42 ]. Nonetheless, the utilization of virtual learning environments encounters challenges such as heightened cognitive load and information disorientation. Consequently, the meticulous design of virtual learning environments to maximize the development and augmentation of college students’ critical thinking capacities becomes paramount.

Limitations and prospects of this study

The conclusions drawn from this study are based on a high-quality randomized controlled trial, significantly elevating the strength of its evidentiary foundation in contrast to cohort studies. This robust evidence, synonymous with evidence-based medicine, lays a foundational bedrock for the prospective application of virtual reality technology within nursing education. Nevertheless, this meta-analysis has certain limitations: (1) The incorporated literature consisted exclusively of randomized controlled trials, albeit a subset of these studies omitted descriptions of the allocation concealment strategy; (2) Disparities in intervention types and assessment methods for outcome indicators among the included studies might have potentially influenced the eventual aggregated outcomes; (3) This investigation limited its scope to Chinese and English literature, potentially introducing an influence on the study outcomes. Consequently, to further establish the efficacy of psychotherapy in treating depression and anxiety among college students, subsequent stages necessitate the design of more rigorous, multicenter, and large-scale randomized controlled trials.

The outcomes of this meta-analysis demonstrate that VR technology can be more effective in improving nursing students’ knowledge of nursing teaching skills, practical nursing teaching aptitude, and academic contentment. Nevertheless, no notable superiority of VR technology was observed in enhancing nursing students’ critical thinking abilities. This could potentially stem from variations in intervention delivery, assessment methodologies, study participants, and research schemes. Consequently, educators must reorient their teaching paradigms, reinforcing the significance of virtual reality technology, and proactively integrating advanced technological tools for educational advancement. Based on the aforementioned points, VR technology stands poised to emerge as a pivotal breakthrough in the future of education, ushering in far-reaching impacts on the evolution of pedagogical methodologies in nursing education.

In conclusion, this study demonstrates the potential of VR technology to enhance nursing education by improving theoretical and practical knowledge as well as academic satisfaction among nursing students. However, the absence of a significant advantage in enhancing critical thinking skills through VR interventions suggests the need for further investigation into the design of VR-based learning environments tailored to fostering critical thinking. Despite the rigorous methodology applied in this study, limitations include variations in intervention types, assessment methods, and subject characteristics across the included studies. To address these limitations, future research should focus on refining VR interventions for nursing education, considering the specific components that effectively promote critical thinking, and conducting multicenter studies with larger sample sizes to provide more robust evidence of VR’s impact on nursing education outcomes.

Data Availability

All data generated or analyzed during this study are included in this published article.

Abbreviations

China National Knowledge Infrastructure

Mean difference

The Preferred Reports for Systematic Reviews and Meta-Analyses

Randomized controlled trial

China Biomedical Literature Service

Standardized mean difference.

• Virtual reality

ou Xiangjun S, Jian H, Hanwu et al. The evolution and prospect of virtual reality technology[J]. J Syst Simul, 2004, (09): 1905–9.

Xiuyu YAO, Hui ZHANG, Xiaopeng HUO, et al. Analysis of the applicability of virtual reality technology applied to basic nursing skills teaching[J]. Research on Medical Teaching in Colleges and Universities. (Electronic Edition). 2022;12(05):7–13.

Google Scholar  

Jarvelainen M, Cooper S, Jones J. Nursing students’ educational experience in regional Australia: reflections on acute events. A qualitative review of clinical incidents [J]. Nurse Educ Pract. 2018;31:188–93.

Article   Google Scholar  

Atli K, Selman WR, Ray A. Virtual reality in neurosurgical education: modernizing the medical classroom [J].Neurosurgery, 2020, 67: 51–51.

Kyaw BM, Saxena N, Posadzki P, et al. Virtual reality for health professions education: systematic review and meta-analysis by the digital health education collaboration [J]. J Med Internet Res. 2019;21(1):e12959.

Yu MH, Yang WY. The application effect of virtual reality technology in the teaching of nursing students in rehabilitation department [J]. China High Med Educ, 2020(9): 103–4.

Zhao J, Lu Y, Zhou F, Mao R, Fei F. Systematic bibliometric analysis of research hotspots and Trends on the application of virtual reality in nursing. Front Public Health. 2022;10:906715. https://doi.org/10.3389/fpubh.2022.906715

Jeeyae C, Elise CT. Faculty Driven Virtual Reality (VR) Scenarios and Students Perception of Immersive VR in Nursing Education: A Pilot Study.[J]. AMIA … Annual Symposium proceedings. AMIA Symposium,2022,2022.

Iwanaga J, Loukas M, Dumont AS et al. A review of anatomy education during and after thecovid-19 pandemic: revisiting traditional and modern methods to achieve future innovation[J].Clinical anatomy, 2021, 34(1): 108–14.

Lange KA, Koch J, Beck A et al. Learning with virtual reality in nursing education - A qualitative research study using the UTAUT Model (Preprint)[J]. JMIR Nursing,2020,3(1).

Ahmed H, Allaf M, Elghazaly H. Covid-19 and medical education [. J] Lancet Infectious Diseases. 2020;20(7):777–8.

Li Y, Li K, Wei W, et al. Critical thinking, emotional intelligence and conflict management styles of medical students: a cross-sectional study [J]. Think Skills Creativity. 2021;40:100799.

Jang S, Vitale JM, Jyung RW, et al. Direct manipulation is better than passive viewing for learning anatomy in a three-dimensional virtual reality environment [. Volume 106. J].Computers & Education; 2017. pp. 150–65.

Yung R, Khoo-Lattimore C. New realities: a systematic literature review on virtual reality and augmented reality in tourism research [J]. Issues in Tourism. 2019;22(17):2056–81.

Harrington CM, Kavanagh DO, Quinlan JF, et al. Development and evaluation of a trauma decision making simulator in oculus virtual reality [J]. Am J Surg. 2018;215(1):42–7.

Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA Group Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.Ann. Intern. Med. 2009;151:264–9. https://doi.org/10.7326/0003-4819-151-4-200908180-00135

Sherman WR, Craig AB. Understanding virtual reality: interface, application, and design [M]. San Mateo, CA: Morgan Kaufmann Publishers Inc; 2002.

Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Sterne JA. The cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:889–93. https://doi.org/10.1136/bmj.d5928

HIGGINS JP, GREEN S. Cochrane handbook for systematic reviews of intervention version 5.1.0 [EB/OL]. [2022-03-13]. https://handbook-5-1.cochrane.org/

Padilha JM, Machado PP, Ribeiro A, Ramos J, Costa P. Clinical virtual Simulation in nursing education: Randomized Controlled Trial. J Med Internet Res. 2019;21(3):e11529. https://doi.org/10.2196/11529

Lee H, Han JW. Development and evaluation of a virtual reality mechanical ventilation education program for nursing students. BMC Med Educ. 2022;22(1):775. https://doi.org/10.1186/s12909-022-03834-5

Al-Mugheed K, Bayraktar N, Al-Bsheish M, AlSyouf A, Aldhmadi BK, Jarrar M, Alkhazali M. (2022). Effectiveness of game-based virtual reality phone application and online education on knowledge, attitude and compliance of standard precautions PloS one, 17(11), e0275130. https://doi.org/10.1371/journal.pone.0275130

Jung AR, Park EA. The effectiveness of learning to Use HMD-Based VR Technologies on nursing students: Chemoport insertion surgery. Int J Environ Res Public Health. 2022;19(8):4823. https://doi.org/10.3390/ijerph19084823

Yuan T, Shen J, Li X. Application of flipped classroom teaching model based on virtual reality animation micro-class in internal medicine nursing[J]. Nurs Res. 2019;33(24):4325–7.

Yu M, Yang M, Ku B, Mann JS. Effects of virtual reality Simulation Program regarding high-risk neonatal infection control on nursing students. Asian Nurs Res. 2021;15(3):189–96. https://doi.org/10.1016/j.anr.2021.03.002

Xiaoyan, Wang, Hu Rongfang,Yan Yilu. Application of virtual simulation technology in obstetric nursing experimental teaching[J]. China High Med Educ,2023(02):44–5.

Chan HY, Chang HC, Huang TW. Virtual reality teaching in chemotherapy administration: Randomised controlled trial. J Clin Nurs. 2021;30(13–14):1874–83. https://doi.org/10.1111/jocn.15701

Ping WangYan, Wenzhen WANG, Fang,WANG Liping CAI, Hongya. CHEN Weiping. Application of immersive virtual reality technology in the experimental teaching of intravenous injection method[J]. China Nurs Manage 2020,20(02):176–80.

Liping Li. The use of virtual reality technology in nursing laboratory teaching [J]. Electron J Clin Med Literature 2017,4(84):16622–3. https://doi.org/10.16281/j.cnki.jocml.2017.84.105

Hongmei Zhao,Ma Di,Liu Hongchun. The application of virtual simulation experimental teaching mode in the teaching of acute and critical care nursing [J]. Mod Hosp 2022,22(10):1608–10.

Nan Cao. Application of simulation teaching mode of virtual reality technology in higher vocational surgical nursing [J]. Health Professions Education 2021,39(08):87–9.

Chavez B, Bayona S et al. Virtual reality in the learning process [C] //Rocha A, Adeli H, Reis L,. Trends and advances in information systemsand technologies. Switzerland: Springer, Cham, 2018:1345–1356.

Forsberg E, Ziegert K, Hult H, et al. Assessing progression of clinical reasoning through virtual patients: an exploratory study [J]. Nurse Educ Pract. 2016;16(1):97–103.

Chan CKY. Exploring an experiential learning project through Kolb’s Learning Theory using a qualitative research method [J]. Eur J Eng Educ. 2012;37(4):405–15.

Woon APN, Mok WQ, Chieng YJS, Zhang HM, Ramos P, Mustadi HB, et al. Effectiveness of virtual reality training in improving knowledge among nursing students: a systematic review, meta-analysis and meta-regression. Nurse Educ Today. 2021;98:104655.

Choi KS. Virtual eeality in nursing: nasogastric tube placement training simulator [J]. Stud Health Technol Inform. 2017;245(23):1298.

Jefferson G, Samah A, Dena M, et al. The acquired critical thinking skills, satisfaction, and self confidence of nursing students and staff nurses through high-fidelity simulation experience[J]. Clin Simul Nurs. 2022;64:24–30.

Azher S. Cervantes Amanda,Marchionni Caroline,Grewal Keerat,Marchand Hugo,Harley Jason M. Virtual Simulation in Nursing Education: Headset Virtual Reality and Screen-based Virtual Simulation Offer A Comparable Experience[J]. Clinical Simulation in Nursing,2023,79.

Chang SC, Hsu TC, Jong M. Integration of the peer assessment approach with a virtual reality design system for learning earth science[J]. Computers and Education. 2020;146:1–15.

Jun’e, Liu. Jingyue. Mobilizing students’ independent learning ability to improve the effectiveness of problem-based learning [J]. Chin Nurs Educ. 2010;07(12):563–5.

Kandi VR, Castronovo F, Brittle P et al. Assessing the impact of a construction virtual reality game on design review skills of construction students[J]. J Architectural Eng 2020,26(4):1–22.

Kang SJ, Hong CM, Lee H. The impact of virtual simulation on critical thinking and self-directed learning ability of nursing students[J]. Clin Simul Nurs 2020,12(49):66–72.

Download references

Acknowledgements

Not applicable.

Funding for this study was provided by the Jining Medical University Practical Teaching Education Research Program Project, China (JYSJ2022A05). Funding body played a role in the design of the study.

Author information

Authors and affiliations.

Nursing Department of Affiliated Hospital of Jining Medical University, 89# Guhuai Road, Rencheng district, Jining City, 272000, Shandong Province, China

Kai Liu, Wei Li, Ting Wang & Yanxue Zheng

Interventional Radiology of Affiliated Hospital of Jining Medical University, 89# Guhuai Road, Rencheng district, 272000, Jining City, Shandong, China

Weiwei Zhang

You can also search for this author in PubMed   Google Scholar

Contributions

All authors conceived and designed the study. ZYX and LK organized and conducted the search and participated in the selection of the relevant articles and performed the quality assessment of the studies. ZWW,WT and LW supervised the project and made the changes and corrections suggested by the reviewers.

Corresponding author

Correspondence to Yanxue Zheng .

Ethics declarations

Ethics approval and consent to participate.

Ethical approval was not required for this meta-analysis.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Liu, K., Zhang, W., Li, W. et al. Effectiveness of virtual reality in nursing education: a systematic review and meta-analysis. BMC Med Educ 23 , 710 (2023). https://doi.org/10.1186/s12909-023-04662-x

Download citation

Received : 23 May 2023

Accepted : 08 September 2023

Published : 28 September 2023

DOI : https://doi.org/10.1186/s12909-023-04662-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Nursing education
• Nursing students
• Meta-analysis

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Research article
• Open access
• Published: 22 March 2021

Exploring the pedagogical design features of the flipped classroom in undergraduate nursing education: a systematic review

• Punithalingam Youhasan   ORCID: orcid.org/0000-0002-3435-7839 1 , 2 ,
• Yan Chen 1 ,
• Mataroria Lyndon 1 &
• Marcus A. Henning 1  

BMC Nursing volume  20 , Article number:  50 ( 2021 ) Cite this article

11k Accesses

41 Citations

1 Altmetric

Metrics details

In recent years, technological advancement has enabled the use of blended learning approaches, including flipped classrooms. Flipped classrooms promote higher-order knowledge application – a key component of nursing education. This systematic review aims to evaluate the empirical evidence and refereed literature pertaining to the development, application and effectiveness of flipped classrooms in reference to undergraduate nursing education.

A PRISMA systematic review protocol was implemented to investigate the literature pertaining to the development, implementation and effectiveness of flipped classroom pedagogy in undergraduate nursing education. Seven databases (Scopus, PsycINFO, CINAHL, ERIC, MEDLINE, Cochrane, Web of Science) were utilised to survey the salient literature. Articles were appraised with respect to their level of evidence, the origin of study, study design, the aims/s of the study, and the key outcomes of the study. A qualitative synthesis was then conducted to summarise the study findings.

The initial search identified 1263 potentially relevant articles. After comprehensively reviewing the initial catchment using several analytical phases, 27 articles were considered for the final review, most of which were conducted in the USA and South Korea. A range of research designs were applied to measure or discuss the outcomes and design features of the flipped classroom pedagogy when applied to undergraduate nursing education. The review indicated that a common operational flipped classroom model involves three key components, namely pre-classroom activities, in-classroom activities and post-classroom activities, guided by two instructional system design principles. The review predominantly identified positive learning outcomes among undergraduate nursing students, after experiencing the flipped classroom, in terms of skills, knowledge and attitudes. However, a few studies reported contrasting findings, possibly due to the incompatibility of the flipped classroom pedagogy with the traditional learning culture.

Conclusions

Current evidence in this systematic review suggests that incorporating the flipped classroom pedagogy could yield positive educational outcomes in undergraduate nursing education. There are promising pedagogical models available for adapting or developing the flipped classroom pedagogy in undergraduate nursing education.

Peer Review reports

Globally, nursing educational institutions are taking steps forward in redesigning their curricula to align them with modern pedagogy to enhance student-centred learning [ 1 ]. This curricular reform is concerned with fulfilling the educational needs of the new technological era, which generates exposure to a variety of information, advanced communication technology, and diverse learning methods [ 2 ]. As a result, blended learning has become part of this curricular reform. Blended learning is a broad pedagogical approach, which encompasses a combination of face-to-face and online teaching to promote student-centred learning [ 3 ]. One of the blended learning innovations is the notion of the flipped classroom [ 4 ], referred to as “a hybrid approach to learning, using technology to move the classroom lecture to homework status and using face-to-face classroom time for interactive learning” [ 5 ]. The rule of thumb of the flipped classroom is redesigning the face-to-face classroom as an interactive learning environment where higher-ordered learning takes place, while providing traditional pedagogical experiences (of transferring basic information) through pre-class learning activities [ 4 , 6 , 7 , 8 ]. As such, pre-class learning materials can be provided to closely represent learning in the traditional face-to-face classroom but being delivered electronically or via online media [ 9 ]. Therefore, pre-learning materials should be accompanied with teachers’ explanation rather than relying on the sole use of pre-class reading materials [ 6 ].

Contemporary nursing care is advancing dramatically due to the need for nursing students to manage problems associated with multiple and complex clinical comorbidities [ 10 ]. It was reported in the literature that nursing students experience difficulties in applying learnt knowledge in clinical practice [ 11 ]. This necessitated the need for nursing curricular implementers to adopt pedagogies like the flipped classroom to ensure that theoretical concepts were explicitly linked to patient care. As such, the flipped classroom is considered as a new educational paradigm for implementing health professions’ education curricula [ 6 , 12 ]. While there are increasing applications of the flipped classroom, there is a dearth of evidence evaluating its impact on student learning and curriculum design in undergraduate nursing education [ 13 , 14 , 15 ]. The empirical evidence to date has predominantly examined the effectiveness of the flipped classroom through students’ satisfaction and academic performance [ 13 ]; however, there is limited evidence to explain the pedagogical design principles of the flipped classroom, which are indispensable to achieving meaningful educational effectiveness [ 16 ]. Thus, the following systematic review aims to describe and evaluate research conducted in the area of designing, developing and implementing the flipped classroom, and appraise the educational impact of the flipped classroom approach when applied to undergraduate nursing education.

This systematic review was performed in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) protocols (Additional File) [ 17 ]. PRISMA is an evidence-based system used to guide reporting in systematic reviews and meta-analyses [ 18 ]. The protocol was registered with the PROSPERO (International prospective register of systematic reviews) ( CRD42020194474 , 16th October 2020).

Systematic literature search

A literature search was conducted utilising seven databases (PsycINFO, CINAHL, ERIC, MEDLINE, Web of Science, Cochrane Library and Scopus) in November 2019. The key concept of the literature search was the term flipped classroom. This term was combined with a range of supplementary key words relevant to nursing education using a PICOS (Population, Intervention, Comparison, Outcomes and Study) framework [ 19 ]. The derived terms were: Population - Undergraduate Nursing Students; Intervention - Flipped Classroom; Comparison - Traditional Classroom; Outcomes - Educational achievements and pedagogical designs; and Study - any original research studies. A search algorithm was created by using keywords with Boolean operators to conduct a literature search in the databases. A sample search strategy in MEDLINE is illustrated in Table  1 .

Study selection

Titles of the manuscripts which were identified in the database search were transferred to a bibliography management programme (Endnote X9, Thomson Reuters, New York) to create a search library and remove duplicates. The resulting studies were independently and systematically reviewed by an author (PY) in accordance with the inclusion criteria (Table  2 ), first by title and then by abstract. Then, full texts of the selected studies were again reviewed by the author (PY) and he made a log of all reviewed studies with reasons for inclusion or exclusion. The log was cross-checked by the other three authors (YC, ML & MAH). Following this, all four authors met at various times to discuss and review all chosen articles. Any disagreements were resolved through discussions within the whole research group until a consensus was reached. Moreover, citations from the selected studies were scrutinised to confirm that all relevant studies were identified.

Data synthesis

The data synthesis was performed using an electronic data extraction table (in Microsoft Excel). The following details were extracted from each reviewed study: name of authors, country, publication year, participants, research design, research procedure, research instrument, analysis of data, key findings and conclusion. The initial data extraction was completed by PY. The extracted data were independently reviewed for accuracy by the other three authors (YC, ML, & MAH), This group confirmed the inter-rater reliability and resolved any outstanding issues, such as data entry errors. Furthermore, if the details from a selected study was inadequate or ambiguous, additional information was obtained from the corresponding author/s of the relevant study. Lastly, an inductive thematic method was used to analyse the extracted (qualitative) data [ 20 ]. This process incorporated a series of inductive stages. First, the extracted data were line-by-line coded by the first author (PY). Then, the codes were crossed checked (by all authors) and clustered under descriptive themes. Finally, the descriptive themes were further condensed into analytical themes to provide an in-depth description regarding the aims of the review. With the exception of the first step, all other steps were conducted in a meeting with the presence of all four authors for establishing inter-rater reliability.

Quality assessment of the selected studies

An evidence hierarchy classification model (Table  3 ) was used to assess the quality of the studies [ 21 , 22 , 23 ]. Each publication included in the data synthesis was then allocated to an evidence hierarchy classification (I to IV). Subsequently, the publication was assigned to the operational ranks as devised by Jensen et al. (2004) [ 22 ]. To maintain the integrity of the quality assessment process, evidence appraisals were independently rated by two authors (PY & MAH). The ratings were presented and discussed amongst all four authors in a meeting to finalize the allocation of category of evidence.

The initial search yielded a total of 1263 hits from the seven databases (PsycINFO = 53, CINAHL = 145, ERIC = 361, MEDLINE = 374, Web of Sci. = 196, Cochrane =10, Scopus = 124). One hundred and sixty duplicates were identified, and 1103 studies were considered for title and abstract screening. In this title and abstract screening, 629 studies were excluded as they were deemed out of scope. The subsequent quota of studies ( n  = 474) was included for assessing the full texts. A list of 104 studies was identified as potentially relevant to the systematic literature review by three authors. Further, this was reduced to a final list of 27 refereed sources after appraisal of the full texts (Fig.  1 ). The key study features of the 27 articles in the evidence synthesis are presented in Table  4 .

Search methodology PRISMA flow diagram

Study characteristics

Study participants.

Participants of the study were defined as undergraduate students who enrolled in the nursing programme. Eleven studies reported the details of students’ academic year of study. Accordingly, the academic year of participating nursing students range from 1st to 4th year. Nevertheless, most of the studies ( n  = 6) were conducted among third-year nursing students (Table 4 ).

Demographic information of the participants, including age and gender, was included in 12 studies. Two further studies only included the age of participants, while an additional study reported only gender. However, 12 studies did not report demographic variables. The majority of the reported participants were females, and the mean age range was from 19 to 31.5 years. Sample size was reported in the 23 studies which, ranged from 14 to 589.

Study setting

The selected studies were primarily conducted in the context of tertiary level nursing education. Most of the studies were conducted in the USA ( n  = 11, 40.7%), followed by South Korea ( n  = 9, 30.3%). Two articles each were found to be published from Mainland China ( n  = 2) and Australia ( n  = 2). One article was published from Norway, Iran and Hong Kong. Moreover, the flipped classroom experiences were reported in reference to a vast range of nursing subjects or courses (Table 4 ).

Methodical quality of studies

According to the evidence hierarchy classification, the majority of the articles ( n  = 19) were IIb ( n  = 5) or above ( n  = 14). The review also included one Ia category evidence [ 44 ]. Interestingly, according to operational ranks, all the articles which were included for the review were clustered into rank A (Table 4 ).

Evidence synthesis on the flipped classroom in nursing education

Qualitative thematic synthesis of findings.

The thematic synthesis revealed 37 codes. The identified codes were clustered into four descriptive themes; namely, knowledge and skills; attitudes and perceptions; flipped classroom (FC) design; and teaching and learning (TL) strategies. The descriptive themes were further specified to two analytical themes for providing profound insights and excelling the context of the present review. The analytical themes revealed were i) the pedagogical structure of the flipped classroom and ii) influence of flipped classroom on nursing students’ learning (Fig.  2 ).

Thematic synthesis

Analytical theme 1: pedagogical structure of the flipped classroom

The systematic review revealed that the flipped classroom is a new educational paradigm in undergraduate nursing education [ 6 , 12 ]. A common operational flipped classroom model (Fig.  4 ) was reported in the selected literature which consisted of three components, namely pre-classroom activities, in-classroom activities and post-classroom activities [ 1 , 15 , 25 , 27 , 28 , 31 , 33 , 35 , 38 , 39 , 40 , 42 , 43 ]. Pre-class activities require the provision of learning material by educators to students [ 25 ]. The learning material is mainly transferred through an online learning management system to students [ 1 , 24 , 25 ], at least one week before the face-to-face classroom session [ 3 , 27 , 28 ]. The pre-learning materials can be delivered in various forms, such as video lecture, narrated PowerPoint, animation (illness scenario) and video demonstrations of the nursing procedure [ 1 , 25 , 36 , 38 ] with further readings [ 33 , 35 , 38 , 42 ]. The duration of the video learning material ranged from 10 to 20 min [ 15 , 27 , 31 ]. Different software were used to create pre-learning materials, such as “Articulate Storyline” [ 3 ], “Explain Everything” [ 27 ], “Camtasia Studio” [ 12 ]. At the end of pre-class activities, an assessment was conducted mainly as quizzes [ 28 , 31 , 35 , 39 ]. Furthermore, students were able to interact with teachers and peers through online dashboards [ 15 , 24 , 25 , 28 , 39 ].

The in-classroom learning environment was designed as an interactive space for applying, analysing and evaluating the pre-learning material [ 1 ]. For in-classroom activities, students were divided into small groups [ 3 , 31 , 36 , 38 , 39 , 40 ] and the reported group size ranged from two to six [ 3 , 25 , 38 , 42 ]. Some studies used quizzes as a diagnostic test at the beginning of the in-class activities [ 3 , 12 , 43 ], followed by several student-centred learning activities [ 3 , 5 , 27 , 28 , 31 , 35 , 36 , 38 ]. Other studies reported that teachers conducted a micro-lecture for summarizing and clarifying complex phenomena [ 12 , 27 , 40 , 42 , 43 ].

Post-class activities continued with a follow-up discussion of the newly learnt concepts or issues which had not been solved in the previous in-class session [ 1 , 25 ]. The follow-up discussions were mainly conducted online [ 1 ]. Post-class tests can be conducted to assess students' learning [ 3 , 42 ]. Finally, at the end of the flipped classroom experience, students completed a survey to evaluate the effectiveness of the flipped classroom [ 12 , 31 ].

In terms of developing a flipped classroom, two studies were identified that investigated instructional system designs. Lee and Park (2018) outlined nine design principles for developing a flipped classroom that could be used in a surgical nursing practicum [ 15 ]. These are illustrated in the Fig.  3 .

Nine design principles for developing flipped classroom [ 15 ]

Oh et al. (2019) used a framework termed the ADDIE model for developing a flipped classroom to teach a nursing informatics course [ 24 ]. This model has been used in terms of its five straightforward steps: Analyze, Design, Develop, Implement, and Evaluate (Fig. 4 ). First, the Analyze step involves the assessment of feasibilities for adopting flipped classroom in terms of current practice, equipped environment, stakeholder’s readiness and nature of the curriculum. The Design phase deals with framing the instructional strategies, such as identifying courses for implementation, defining the operational procedures, lesson planning, choosing assessment instruments, designing the user interface and choosing the audio-visual designs. The Development phase starts with the production of teaching-learning material for the flipped classroom. In addition, an instrument is developed for measuring the effectiveness of the flipped classroom on students’ learning. The Implementation phase requires participants to receive the flipped classroom. The last step of the ADDIE method is Evaluation. The main aims of this phase are to gather feedback from participants and assess the educational improvement of the learners to quantify the effectiveness of the intervention and identify the way forwards for future improvement [ 24 ].

“ADDIE” model and Flipped Classroom Operational Model

Influence of flipped classroom on nursing students’ learning

Several studies addressed the effect of flipped classroom learning on the students’ knowledge and skill. Oh et al. (2019) revealed that integrating flipped classroom led to a significant improvement in nursing students’ test scores [ 24 ]. The same impact was observed in another seven studies [ 5 , 28 , 29 , 31 , 32 , 36 , 42 ]. In contrast, four studies concluded that flipped classroom learning does not influence students’ assessment performance [ 30 , 34 , 35 , 41 ].

Six studies reported the influence of flipped classroom on nursing students’ skill development. Kim and Jang (2017) revealed that students’ clinical performance evaluation score increased after 10 weeks of a flipped classroom intervention [ 31 ]. Similarly, nursing students’ core competencies in clinical practicum increased after the flipped classroom experience in a clinical setting [ 27 ]. The above two findings were endorsed by a meta-analysis of 22 randomised controlled trials, reporting that flipped classroom improved nursing students’ skill competence [ 44 ]. The effect was further confirmed by two studies indicating that the flipped classroom approach increased nursing students’ confidence in performing core skills [ 25 , 40 ]. Kim et al. (2019) reported that nursing students’ patient safety competency was significantly increased after the flipped classroom experience [ 28 ]. In terms of problem-solving skill, Lee and Park (2018) concluded that nursing students who received the flipped classroom showed significant improvements in problem-solving skills [ 15 ]. A study investigated communication skills, reported that the outcome of therapeutic communication was significantly increased among the nursing students, after attending the flipped classroom on the mental health nursing practicum [ 27 ].

There were several noticeable attitudinal changes among nursing students due to the flipped classroom, namely satisfaction, motivation, engagement, confidence, self-directedness, enjoyment, and critical thinking. In terms of satisfaction, four studies reported that flipped classroom learning increased nursing students’ satisfaction [ 1 , 25 , 31 , 37 ]. In contrast, one study noted that nursing students expressed more satisfaction with the traditional lecture-based learning model than flipped classroom learning [ 30 ]. In addition, a study reported that nursing students’ satisfaction plummeted at the initial period with the introduction of the flipped classroom [ 36 ]. Moreover, nursing students with kinesthetic learning styles were satisfied with the flipped classroom while learners classified as having a preference for visual and auditory stimuli preferred traditional teaching methods [ 38 ].

In relation to self-directed learning, four studies reported that flipped classroom pedagogy enhanced self-directed learning among nursing students [ 1 , 26 , 30 , 40 ]. Self-goal setting ability was significantly increased among nursing students who attended a flipped classroom [ 15 ]. Regarding nursing students’ motivation, two studies indicated that the flipped classroom enhanced learning motivation [ 1 , 26 ]. In reference to nursing students' learning engagement, the flipped classroom was recognised as an active learning method which enhanced learning engagement [ 35 ]. In addition, flipped classroom approaches improved nursing students’ cooperative spirit and teamwork, which increased their interest to engage in learning [ 44 ].

Four studies found the flipped classroom an enjoyable way of learning in nursing education [ 1 , 33 , 35 , 44 ]. It is interesting to note that nursing students enjoyed viewing the video lecture more than the live lecture [ 33 ]. In terms of critical thinking, flipped classrooms increased nursing students’ critical thinking [ 3 , 42 ]. Moreover, the flipped classroom enabled nursing students’ ability to think deeply and analyse the problem [ 43 , 44 ].

This systematic review explored and evaluated the flipped classroom in the context of undergraduate nursing education. In particular, the systematic review addressed two main aspects – one focusing on the design and development of flipped classroom pedagogy in undergraduate nursing education and the other evaluating the impact of the flipped classroom on undergraduate nursing students’ learning.

The systematic review identified 27 studies that investigated the flipped classroom experience among undergraduate nursing students. According to the evidence hierarchy classification model [ 21 , 22 ], most of the selected studies reached the evidence category of IIb or above and all of them achieved operational rank “A”, indicating a catchment of high quality papers. A variety of methodologies, including educational measures were used to determine the impact of the flipped classroom on undergraduate nursing students’ learning. As the studies varied significantly, it is not easy to perform a direct comparison between studies due to the degree of heterogeneity. Nevertheless, the results of the selected studies revealed that a common operational pedagogical structure (Fig. 4 ) was generally utilised regardless of instructional system designing principles (except for two studies) and the flipped classroom resulted in positive learning outcomes among undergraduate nursing students.

Taking the flipped classroom design into consideration, studies investigated the flipped classroom design in reference to both micro and macro levels [ 24 , 45 ]. The micro level concerns developing flipped classroom pedagogy for a session or topics [ 45 , 46 ]. In contrast, the macro level involves instructional system design at the curriculum or course level [ 45 , 47 ]. Most of the selected studies included in this review examined the flipped classroom at the micro level. Interestingly, the three-step flipped classroom operational model (Fig. 4 ) describes the flipped classroom design at the micro-level. It was noted that the common operational model was utilised in different forms. For example, Oh et al. (2019) used the basic operational model in eight steps which is called the “C-REVERSE” design with the use of film clips [ 24 ] and “flipped-mastery classroom model” was used in the South Korean clinical nursing practicum curriculum [ 25 ]. However, the existing findings emphasize that the benefits of the flipped classroom did not eventuate based on sticking purely to the common operational model, but rather caused by the logical connections between the different steps [ 45 , 48 , 49 ].

Some studies included the pre-classroom activities and post-classroom activities under a common cluster of online-learning phase and in-classroom activities labelled as face-to-face learning phase [ 1 , 39 , 43 , 45 , 50 ]. Three important concerns were reported for developing the online learning phase [ 45 , 46 , 51 ] through evaluating the: (i) physical feature of the video or online lectures which includes duration, pacing and quality of audio; (ii) content feature of the video or online lecture such as appropriate provision of the online portion, clarity and interactivity; and (iii) logistic feature of the video or online lecture namely formative assessment, timetabling and follow-up activities. Designing the face-to-face learning phase is crucial because it is the core part of the flipped classroom [ 45 ]. The current review suggests four cardinal activities of the in-class activities, namely dividing students into small groups, conducting a diagnostic assessment, micro-teaching, and continuing integrative student-centred instructions. Furthermore, it has been suggested that the face-to-face learning phase should include: (i) introductory tasks such as mini-lecture and authenticating quizzes; (ii) interactive learning activities which are aligned with the intended learning outcomes; and (iii) well established ground rules and learning culture [ 45 , 46 , 52 , 53 ].

In terms of macro-level design, the review identified that the ADDIE model created a framework for designing the flipped classroom for undergraduate nursing students [ 24 ]. The ADDIE model has been recognized as effective, systematic and efficient in designing the flipped classroom in nursing education [ 54 , 55 ]. Moreover, the ADDIE model has achieved acceptance in diverse fields [ 56 , 57 ]. The ADDIE model proposes five straightforward steps when developing the pedagogical strategies used ensure curriculum planners and implementers reach the ‘appropriate destination’ [ 56 ]. In addition, the review traced the design principles of the flipped classroom. Lee and Park (2018) adopted nine design principles (Fig. 3 ) for developing flipped classrooms in reference to the surgical nursing practicum [ 15 ]. Kim et al., (2014) proposed the flipped classroom design principles for enforcing student-centred learning through four key variables, namely cognitive presence, social presence, teaching presence, and learner presence [ 58 ].

In reviewing the impact of the flipped classroom on undergraduate nursing students’ learning, positive outcomes were reported in many studies included in this review. More specifically, nursing students’ knowledge, skills and attitudes were improved by the flipped classroom learning, in terms of assessment performance, performing core skills, problem-solving, communication, critical thinking, self-directedness, motivation, engagement, confidence, satisfaction, and joyful learning. Besides, the notions of positive outcomes were reported among students from other discipline such as dentistry, medicine, pharmacy [ 59 , 60 , 61 , 62 , 63 ]. It was reported from the literature that two main explanations contributed to the positive learning outcomes. Firstly, unimpeded access to the pre-classroom learning materials enabled nursing students to learn in their preferred place, pace and time. Specifically, the pre-recorded video lecture was used as the main pre-classroom learning material. The nursing students who watched the video lectures developed a better understanding of learning concepts [ 6 ]. Secondly, in-classroom activities were designed as an interactive and student-centred environment which provided greater opportunity to apply the learned concepts into practice [ 1 ]. On the other hand, some studies still favoured the traditional lecture-based learning [ 30 , 41 ]. This may be due to the preference for behaviouristic learning in higher education. Overall, the findings so far seem to suggest that we still have mixed results on whether flipped classroom increases test scores; however, there seems to be strong evidence to suggest that flipped classroom can increase student motivation, satisfactory, and critical thinking.

These findings present two important implications for developing and implementing the flipped classroom in undergraduate nursing education. Firstly, contextual compatibility is more important for the success and sustainability of a pedagogical model. Thus, it is essential to follow an instructional system design at the macro-level to develop flipped pedagogy rather than using its’ operational structures alone at the micro-level. The review identified the ADDIE model and the three-step operational model (Fig. 4 ) for fostering flipped classroom at the macro and micro level, respectively. Secondly, it was noted that the flipped classroom resulted in positive learning outcomes among nursing students. This outcome may be optimised by balancing the workload of pre-, in-, and post-class activities at the micro level, rather than providing more emphasis on one phase. Furthermore, a study reported that the flipped classroom was not welcome by stakeholders during the introduction phase [ 36 ]. Consequently, the solidity of the flipped classroom intervention relies on the constant and stable plan of implementation.

This review could have limitations derived from the heterogeneity of study designs. Apart from meta-analysis, randomised controlled studies, and quasi-experimental studies, we also included several non-experimental descriptive studies to cover the range of available evidence. This heterogenous sample of studies does not permit further probing of the evidence, such a meta-analysis of the study outcomes; however, our sample represents the commonly-used and ethical research methods in educational research and provides a starting point for generating higher levels of evidence. Moreover, the reported findings are mainly from the United States of America and South Korea, which are likely well-resourced settings. Consequently, there may be cultural and regional bias in these studies like ethnocentrism, available resources and educational system. Thus, future research could be conducted in other settings, including low and middle-income countries, to strengthen the evidence base.

The evidence cited in this systematic review suggests that incorporating the flipped classroom pedagogy probably yields promising positive educational outcomes in undergraduate nursing education. The majority of the studies utilized a common operational flipped classroom structure as pre-classroom, in-classroom and post-classroom. Furthermore, there are promising instructional system design models available for adapting or developing a flipped classroom. Practical implications of the review are considering contextual compatibility and providing equal importance to all three phases of flipped classroom for augmenting the educational outcomes. In addition, the feasibilities of developing the flipped classroom in a limited-resourced setting are still inconclusive. Therefore, future research should consider developing and implementing flipped classrooms for the limited-resourced undergraduate nursing educational environment by using a compatible instructional system designing model.

Availability of data and materials

Not applicable.

Abbreviations

Preferred Reporting Items for Systematic Reviews and Meta-Analysis

Flipped Classroom

Teaching and Learning

Analyse, Design, Develop, Implement, and Evaluate

Oh J, Kim SJ, Kim S, Vasuki R. Evaluation of the effects of flipped Learning of a nursing informatics course. J Nurs Educ. 2017;56(8):477–83.

Article   PubMed   Google Scholar  

Benner P. Educating nurses: a call for radical transformation-how far have we come? J Nurs Educ. 2012;51(4):183–4.

Dehghanzadeh S, Jafaraghaee F. Comparing the effects of traditional lecture and flipped classroom on nursing students’ critical thinking disposition: a quasi-experimental study. Nurse Educ Today. 2018;71:151–6.

Lee Y, Martin KI. The flipped classroom in ESL teacher education: an example from CALL. Educ Inf Technol. 2020;25(4):2605–33.

Article   Google Scholar  

Missildine K, Fountain R, Summers L, Gosselin K. Flipping the classroom to improve student performance and satisfaction. J Nurs Educ. 2013;52(10):597–9.

Hew KF, Lo CK. Flipped classroom improves student learning in health professions education: a meta-analysis. BMC Med Educ. 2018;18(1):38.

Article   PubMed   PubMed Central   Google Scholar  

Persky AM, McLaughlin JE. The flipped classroom – from theory to practice in health professional education. Am J Pharm Educ. 2017;81(6). https://www.ajpe.org/content/ajpe/81/6/118.full.pdf .

FLN. Definition of Flipped Learning Flipped Learning Network 2014 [09.03.2020]. Available from: https://flippedlearning.org/definition-of-flipped-learning/

Love B, Hodge A, Corritore C, Ernst DC. Inquiry-based Learning and the flipped classroom model. PRIMUS. 2015;25(8):745–62.

Njie-Carr VPS, Ludeman E, Lee MC, Dordunoo D, Trocky NM, Jenkins LS. An integrative review of flipped classroom teaching models in nursing education. J Prof Nurs. 2017;33(2):133–44.

Hashemiparast M, Negarandeh R, Theofanidis D. Exploring the barriers of utilizing theoretical knowledge in clinical settings: a qualitative study. Int J Nurs Sci. 2019;6(4):399–405.

PubMed   PubMed Central   Google Scholar  

Hoover CA, Dinndorf-Hogenson GA, Peterson JL, Tollefson BR, Berndt JL, Laudenbach N. Flipped classroom: do students perceive readiness for advanced discussion? J Nurs Educ. 2018;57(3):163–5.

Bernard JS. The Flipped Classroom: Fertile Ground for Nursing Education Research. Int J Nurs Educ Scholar. 2015;12(1):99.

Betihavas V, Bridgman H, Kornhaber R, Cross M. The evidence for ‘flipping out’: a systematic review of the flipped classroom in nursing education. Nurse Educ Today. 2016;38:15–21.

Lee MK, Park BK. Effects of flipped Learning using online materials in a surgical nursing practicum: a pilot stratified group-randomized trial. Healthc Inform Res. 2018;24(1):69–78.

Morrison GR, Ross SM, Kalman HK. Designing effective instruction, 7th edition. Wiley Global Education; 2012.

Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med. 2009;6(7):e1000097.

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1.

Amir-Behghadami M, Janati A. Population, intervention, comparison, outcomes and study (PICOS) design as a framework to formulate eligibility criteria in systematic reviews. Emerg Med J. 2020;37(6):387.

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8(1):45.

Eccles M, Freemantle N, Mason J. Using systematic reviews in clinical guideline development. In: Systematic Reviews in Health Care.2001. p. 400-9.

Jensen LS, Merry AF, Webster CS, Weller J, Larsson L. Evidence-based strategies for preventing drug administration errors during anaesthesia. Anaesthesia. 2004;59(5):493–504.

Article   CAS   PubMed   Google Scholar  

Webster CS, Luo AY, Krägeloh C, Moir F, Henning M. A systematic review of the health benefits of tai chi for students in higher education. Prev Med Rep. 2016;3:103–12.

Oh J, Kim SJ, Kim S, Kang KA, Kan J, Bartlett R. Development and evaluation of flipped learning using film clips within a nursing informatics course. Jpn J Nurs Sci. 2019;16(4):385–95.

Kim HS, Kim MY, Cho MK, Jang SJ. Effectiveness of applying flipped learning to clinical nursing practicums for nursing students in Korea: A randomized controlled trial. Int J Nurs Pract. 2017;23:5.

Ya-Qian L, Yu-Feng L, Meng-Jie L, Peng-Xi L, Julie T, Li-Na M, et al. Effectiveness of the flipped classroom on the development of self-directed learning in nursing education: a meta-analysis. Front Nurs. 2018;5(4):317–29.

Im S, Jang SJ. Effects of a clinical practicum using flipped Learning among undergraduate nursing students. J Nurs Educ. 2019;58(6):354–6.

Kim YM, Yoon YS, Hong HC, Min A. Effects of a patient safety course using a flipped classroom approach among undergraduate nursing students: a quasi-experimental study. Nurse Educ Today. 2019;79:180–7.

Maxwell KL, Wright VH. Evaluating the effectiveness of two teaching strategies to improve nursing Students' knowledge, skills, and attitudes about quality improvement and patient safety. Nurs Educ Perspect. 2016;37(5):291–2.

Holman R, Hanson AD. Flipped Classroom Versus Traditional Lecture: Comparing Teaching Models in Undergraduate Nursing Courses. Nurs Educ Perspect. 2016;37:6.

Kim H, Jang Y. Flipped Learning with simulation in undergraduate nursing education. J Nurs Educ. 2017;56(6):329–36.

Geist MJ, Larimore D, Rawiszer H, Sager AWA. Flipped versus traditional instruction and achievement in a baccalaureate nursing pharmacology course. Nurs Educ Perspect. 2015;36(2):114–5.

Dabney BW, Mitchell R. Flipping an undergraduate Gerontological nursing course: student perceptions. Nurs Educ Perspect. 2017;38(6):340–1.

El-Banna MM, Whitlow M, McNelis AM. Flipping around the classroom: accelerated bachelor of science in nursing students' satisfaction and achievement. Nurse Educ Today. 2017;56:41–6.

Simpson V, Richards E. Flipping the classroom to teach population health: increasing the relevance. Nurse Educ Pract. 2015;15(3):162–7.

Greenwood VA, Mosca C. Flipping the nursing classroom without flipping out the students. Nurs Educ Perspect. 2017;38(6):342–3.

Saunders A, Green R, Cross M. Making the most of person-centred education by integrating flipped and simulated teaching: an exploratory study. Nurse Educ Pract. 2017;27:71–7.

Green RD, Schlairet MC. Moving toward heutagogical learning: illuminating undergraduate nursing students' experiences in a flipped classroom. Nurse Educ Today. 2017;49:122–8.

Bingen HM, Steindal SA, Krumsvik R, Tveit B. Nursing students studying physiology within a flipped classroom, self-regulation and off-campus activities. Nurse Educ Pract. 2019;35:55–62.

Cho M-K, Kim MY. Outcomes and influential factors applying flipped learning methods in a clinical adult nursing practicum. Int J Nurs Pract. 2019;25(2):e12724.

Harrington SA, Bosch MV, Schoofs N, Beel-Bates C, Anderson K. Quantitative Outcomes for Nursing Students in a Flipped Classroom. Nurs Educ Perspect. 2015;36:3.

Park EO, Park JH. Quasi-experimental study on the effectiveness of a flipped classroom for teaching adult health nursing. Jpn J Nurs Sci. 2018;15(2):125–34.

Hanson J. Surveying the experiences and perceptions of undergraduate nursing students of a flipped classroom approach to increase understanding of drug science and its application to clinical practice. Nurse Educ Pract. 2016;16(1):79–85.

Xu P, Chen Y, Nie W, Wang Y, Song T, Li H, et al. The effectiveness of a flipped classroom on the development of Chinese nursing students' skill competence: a systematic review and meta-analysis. Nurse Educ Today. 2019;80:67–77.

Lee J, Lim C, Kim H. Development of an instructional design model for flipped learning in higher education. Educ Technol Res Dev. 2017;65(2):427–53.

Smith CM, McDonald K. The flipped classroom for professional development: part II. Making podcasts and videos. J Contin Educ Nurs. 2013;44(11):486–7.

Strayer JF. How learning in an inverted classroom influences cooperation, innovation and task orientation. Learn Environ Res. 2012;15(2):171–93.

Garrison DR, Kanuka H. Blended learning: uncovering its transformative potential in higher education. Internet High Educ. 2004;7(2):95–105.

Nuray G, Ercan K, Ozden MY. Design of a blended learning environment: Considerations and implementation issues. Aust J Educ Technol. 2013;29:1.

Google Scholar  

Herreid CF, Schiller NA. Case studies and the flipped classroom. J Coll Sci Teach. 2013;42(5):62–6.

Arnold I. John Hattie: visible learning: a synthesis of over 800 meta-analyses relating to achievement. Int Rev Educ. 2011;57(1):219.

Roehl A, Reddy SL, Shannon GJ. The flipped classroom: an opportunity to engage millennial students through active Learning. J Fam Consum Sci. 2013;105(2):44–9.

Talbert R. Learning MATLAB in the Inverted Classroom. 2012 ASEE Annual Conference & Exposition; San Antonio, Texas. San Antonio ASEE Conferences; 2012. p. 25.883.1–25..19.

Hsu TC, Lee-Hsieh J, Turton MA, Cheng SF. Using the ADDIE model to develop online continuing education courses on caring for nurses in Taiwan. J Contin Educ Nurs. 2014;45(3):124–31.

Lu SC, Cheng YC, Chan PT. Using ADDIE model to develop a nursing information system training program for new graduate nurse. Stud Health Technol Inform. 2016;225:969–70.

PubMed   Google Scholar  

Cheung L. Using the ADDIE model of instructional design to teach chest radiograph interpretation. J Biomed Educ. 2016:9502572.

Reinbold S. Using the ADDIE model in designing library instruction. Med Ref Serv Quart. 2013;32(3):244–56.

Kim MK, Kim SM, Khera O, Getman J. The experience of three flipped classrooms in an urban university: an exploration of design principles. Internet High Educ. 2014;22:37–50.

Bakr MM, Massey WL, Massa HM. Flipping a dental anatomy course: a retrospective study over four years. Educ Res Int. 2016:7097398.

Galway LP, Corbett KK, Takaro TK, Tairyan K, Frank E. A novel integration of online and flipped classroom instructional models in public health higher education. BMC Med Educ. 2014;14(1):181.

Liebert CA, Lin DT, Mazer LM, Bereknyei S, Lau JN. Effectiveness of the Surgery Core Clerkship Flipped Classroom: a prospective cohort trial. Am J Surg. 2016;211(2):451–7 e1.

McLaughlin JE, Griffin LM, Esserman DA, Davidson CA, Glatt DM, Roth MT, et al. Pharmacy student engagement, performance, and perception in a flipped satellite classroom. Am J Pharm Educ. 2013;77(9):196.

Morton DA, Colbert-Getz JM. Measuring the impact of the flipped anatomy classroom: The importance of categorizing an assessment by Bloom's taxonomy. Anat Sci Educ. 2017;10(2):170–5.

Download references

Acknowledgements

We would like to acknowledge Anne Wilson for her contribution in developing the search algorithm.

No grant was obtained for this systematic literature review.

Author information

Authors and affiliations.

Centre for Medical and Health Sciences Education, Faculty of Medical and Health Sciences, The University of Auckland, Auckland, New Zealand

Punithalingam Youhasan, Yan Chen, Mataroria Lyndon & Marcus A. Henning

Department of Medical Education & Research, Faculty of Health-Care Sciences, Eastern University, Sri Lanka, Batticaloa, Sri Lanka

Punithalingam Youhasan

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to the design of the study and created the search strategy. PY conducted the literature search, screened the titles and abstracts, extracted, analysed, and interpreted the data, drafted the initial manuscript. YC, ML, and MAH critically evaluated the search strategy, contributed to the selection of articles, and made significant contributions to the writing of the manuscript. All authors reviewed and approved the final version of the manuscript and agreed to be accountable for all aspects of the work. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Punithalingam Youhasan .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Youhasan, P., Chen, Y., Lyndon, M. et al. Exploring the pedagogical design features of the flipped classroom in undergraduate nursing education: a systematic review. BMC Nurs 20 , 50 (2021). https://doi.org/10.1186/s12912-021-00555-w

Download citation

Received : 08 October 2020

Accepted : 24 February 2021

Published : 22 March 2021

DOI : https://doi.org/10.1186/s12912-021-00555-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Flipped classroom
• Blended Learning
• Inverted classroom
• Nursing education
• Systematic review
• Design principles

BMC Nursing

ISSN: 1472-6955

• General enquiries: [email protected]

• Subscribe to journal Subscribe
• Get new issue alerts Get alerts

Secondary Logo

Journal logo.

Colleague's E-mail is Invalid

Your message has been successfully sent to your colleague.

Save my selection

The Systematic Review

An overview.

Aromataris, Edoardo PhD; Pearson, Alan PhD, RN

Edoardo Aromataris is the director of synthesis science at the Joanna Briggs Institute in the School of Translational Health Science, University of Adelaide, South Australia. Alan Pearson is the former executive director and founder of the Joanna Briggs Institute. Contact author: Edoardo Aromataris, [email protected] . The authors have disclosed no potential conflicts of interest, financial or otherwise.

The Joanna Briggs Institute aims to inform health care decision making globally through the use of research evidence. It has developed innovative methods for appraising and synthesizing evidence; facilitating the transfer of evidence to health systems, health care professionals, and consumers; and creating tools to evaluate the impact of research on outcomes. For more on the institute's approach to weighing the evidence for practice, go to http://joannabriggs.org/jbi-approach.html .

This article is the first in a new series on systematic reviews from the Joanna Briggs Institute, an international collaborative supporting evidence-based practice in nursing, medicine, and allied health fields. The purpose of the series is to show nurses how to conduct a systematic review—one step at a time. This first installment provides a synopsis of the systematic review as a scientific exercise, one that influences health care decisions.

This first article in a new series from the Joanna Briggs Institute provides a synopsis of the systematic review as a scientific exercise, and introduces nurses to the steps involved in conducting one.

Research in the health sciences has provided all health care professions, including nursing, with much new knowledge to inform the prevention of illness and the care of people with ill health or trauma. As the body of research has grown, so too has the need for rigorous syntheses of the best available evidence.

Literature reviews have long been a means of summarizing and presenting overviews of knowledge, current and historical, derived from a body of literature. They often make use of the published literature; generally, published papers cited in a literature review have been subjected to the blind peer-review process (a hallmark of most scientific periodicals). The literature included in a literature review may encompass research reports that present data, as well as conceptual or theoretical literature that focuses on a concept. 1

An author may conduct a literature review for a variety of reasons, including to 1

• present general knowledge about a topic.
• show the history of the development of knowledge about a topic.
• identify where evidence may be lacking, contradictory, or inconclusive.
• establish whether there is consensus or debate on a topic.
• identify characteristics or relationships between key concepts from existing studies relevant to the topic.
• justify why a problem is worthy of further study.

All of these purposes have been well served by a “traditional” or “narrative” review of the literature. Traditional literature reviews, though useful, have major drawbacks in informing decision making in nursing practice. Predominantly subjective, they rely heavily on the author's knowledge and experience and provide a limited, rather than exhaustive, presentation of a topic. 2 Such reviews are often based on references chosen selectively from the evidence available, resulting in a review inherently at risk for bias or systematic error. Traditional literature reviews are useful for describing an issue and its underlying concepts and theories, but if conducted according to no stated methodology, they are difficult to reproduce—leaving the findings and conclusions resting heavily on the insight of the authors. 1, 2 In many cases, the author of the traditional review discusses only major ideas or results from the studies cited rather than analyzing the findings of any single study.

With the advent of evidence-based health care some 25 years ago, nurses and other clinicians have been expected to refer to and rely on research evidence to inform their decision making. The need for evidence to support clinical practice is constantly on the rise because of advances that continually expand the technologies, drugs, and other treatments available to patients. 3 Nurses must often decide which strategies should be implemented, yet comparisons between available options may be difficult to find because of limited information and time, particularly among clinical staff. Furthermore, interpreting research findings as presented in scientific publications is no easy task. Without clear recommendations for practice, it can be difficult to use evidence appropriately; it requires some knowledge of statistics and in some cases extensive knowledge or experience in how to apply the evidence to the clinical setting. 4 Also, many health care devices and drugs come with difficult-to-understand claims of effectiveness. 5

As a result of research, the knowledge on which nursing care is based has changed at a rapid pace. This inexorable progress means that nurses can access biomedical databases containing millions of citations pertinent to health care; these databases are growing at a phenomenal rate, with tens of thousands of publications added every year. The volume of literature is now too vast for nurses and other health care professionals to stay on top of. 3 Furthermore, not all published research is of high quality and reliable; on the contrary, many published studies have used inappropriate statistical methods or have otherwise been poorly conducted.

Such issues affecting research quality can make for research findings that are contradictory or inconclusive. Similarly, using the results of an individual study to inform clinical decision making is not advisable. When compared with other research on the topic, a study may be at risk for bias or systematic error. 5 Therefore, it can be difficult for nurses to know which studies from among the multitude available should be used to inform the decisions they make every day. As a result, reviews of the literature have evolved to become an increasingly important means by which data are collected, assessed, and summarized. 5-7

THE SYSTEMATIC REVIEW

Since the traditional literature review lacks a formal or reproducible means of estimating the effect of a treatment, including the size and precision of the estimate, 2, 7 a considerably more structured approach is needed. The “systematic review,” also known as the “research synthesis,” aims to provide a comprehensive, unbiased synthesis of many relevant studies in a single document. 2, 7, 8 While it has many of the characteristics of a literature review, adhering to the general principle of summarizing the knowledge from a body of literature, a systematic review differs in that it attempts to uncover “all” of the evidence relevant to a question and to focus on research that reports data rather than concepts or theory. 3, 9

As a scientific enterprise, a systematic review will influence health care decisions and should be conducted with the same rigor expected of all research. Explicit and exhaustive reporting of the methods used in the synthesis is also a hallmark of any well-conducted systematic review. Reporting standards similar to those produced for primary research designs have been created for systematic reviews. The PRISMA statement, or Preferred Reporting Items for Systematic Reviews and Meta-Analyses, provides a checklist for review authors on how to report a systematic review. 10 Ultimately, the quality of a systematic review, and the recommendations drawn from it, depends on the extent to which methods are followed to minimize the risk of error and bias. For example, having multiple steps in the systematic review process, including study selection, critical appraisal, and data extraction conducted in duplicate and by independent reviewers, reduces the risk of subjective interpretation and also of inaccuracies due to chance error affecting the results of the review. Such rigorous methods distinguish systematic reviews from traditional reviews of the literature.

The characteristics of a systematic review are well defined and internationally accepted. The following are the defining features of a systematic review and its conduct:

• clearly articulated objectives and questions to be addressed
• inclusion and exclusion criteria, stipulated a priori (in the protocol), that determine the eligibility of studies
• a comprehensive search to identify all relevant studies, both published and unpublished
• appraisal of the quality of included studies, assessment of the validity of their results, and reporting of any exclusions based on quality
• analysis of data extracted from the included research
• presentation and synthesis of the findings extracted
• transparent reporting of the methodology and methods used to conduct the review

Different groups worldwide conduct systematic reviews. The Cochrane Collaboration primarily addresses questions on the effectiveness of interventions or therapies and has a strong focus on synthesizing evidence from randomized controlled trials (RCTs) (see http://handbook.cochrane.org ). 9 Other groups such as the Centre for Reviews and Dissemination at the University of York ( http://bit.ly/1g9WoCq ) and the Joanna Briggs Institute ( http://joannabriggs.org ) include other study designs and evidence derived from different sources in their systematic reviews. The Institute of Medicine issued a report in 2011, Finding What Works in Health Care: Standards for Systematic Reviews , which makes recommendations for ensuring “objective, transparent, and scientifically valid reviews” (see http://bit.ly/1cRIAg7 ).

How systematic reviews are conducted may vary; the methods used will ultimately depend on the question being asked. The approach of the Cochrane Collaboration is almost universally adopted for a clear-cut review of treatment effectiveness. However, specific methods used to synthesize qualitative evidence in a review, for example, may depend on the preference of the researchers, among other factors. 7 The steps for conducting a systematic review will be addressed below and in greater detail throughout this series.

Review question and inclusion criteria. Systematic reviews ideally aim to answer specific questions, rather than present general summaries of the literature on a topic of interest. 5, 8 A systematic review does not seek to create new knowledge but rather to synthesize and summarize existing knowledge, and therefore relevant research must already exist on the topic. 3, 5 Deliberation on the question occurs as a first step in developing the review protocol. 5, 7 Nurses accustomed to evidence-based practice and database searching will be familiar with the PICO mnemonic (Population, Intervention, Comparison intervention, and Outcome measures), which helps in forming an answerable question that encompasses these concepts to aid in the search. 3, 8, 11 (The art of formulating the review question will be covered in the second article of this series.)

Ideally, the review protocol is developed and published before the systematic review is begun. It details the eligibility of studies to be included in the review (based on the PICO elements of the review question) and the methods to be used to conduct the review. Adhering to the eligibility criteria stipulated in the review protocol ensures that studies selected for inclusion are selected based on their research method, as well as on the PICO elements of the study, and not solely on the study's findings. 3 Conducting the review in such a fashion limits the potential for bias and reduces the possibility of altering the focus or boundaries of the review after results are seen. In addition to the PICO elements, the inclusion criteria should specify the research design or types of studies the review aims to find and summarize, such as RCTs when answering a question on the effectiveness of an intervention or therapy. 9 Stipulating “study design” as an extra element to be included as part of the inclusion criteria changes the standard PICO mnemonic to PICOS.

Searching for studies can be a complex task. The aim is to identify as many studies on the topic of interest as is feasible, and a comprehensive search strategy must be developed and presented to readers. 3, 10 A strategy that increases in complexity is common, starting with an initial search of major databases, such as MEDLINE (accessed through PubMed) and the Cumulative Index to Nursing and Allied Health (CINAHL), using keywords derived from the review question. This preliminary search helps to identify optimal search terms, including further keywords and subject headings or indexing terms, which are then used when searching all relevant databases. Finally, a manual search is conducted of the reference lists of all retrieved papers to identify any studies missed during the database searches. The search should also target unpublished studies to help minimize the risk of publication bias 3, 5 —a reality that review authors have to acknowledge. It arises because researchers are more likely to submit for publication positive rather than negative findings of their research, and scientific journals are inclined to publish studies that show a treatment's benefits. Therefore, relying on findings only from published studies may result in an overestimation of the benefits of an intervention. To date, locating unpublished studies has been difficult, but resources for locating this “gray” literature are available and increasing in sophistication. For example, Web search engines can search across many governmental and organizational sites simultaneously. Similarly, there are databases that index graduate theses and doctoral dissertations, abstracts of conference proceedings, and reports that aren't commercially published. Contacting experts in the field may also yield otherwise difficult-to-obtain information. Finally, studies published in languages other than English should be included, if possible, despite the added cost and complexity of doing so. (The art of searching will be addressed in the third paper in this series.)

Study selection and critical appraisal. The PICO elements can aid in defining the inclusion criteria used to select studies for the systematic review. The inclusion criteria place the review question in a practical context and act as a clear guide for the review team as they determine which studies should be included. 3 This step is referred to as study selection . 8 Once it's determined which studies should be included, their quality must be assessed during the step of critical appraisal . (Both of these steps will be further addressed in the fourth paper in this series.)

During study selection , reviewers look to match the studies found in the search to the review's inclusion criteria—that is, they identify those studies that were conducted in the correct population, use interventions of interest, and record the predetermined and relevant outcomes. 3 The optimal research design for answering the review question is also determined. For example, for a systematic review evaluating the effectiveness of an intervention, the most reliable evidence is thought to come from RCTs, which allow the inference of causal associations between the intervention and outcome, rather than from other study designs such as the cohort study, which lacks randomization and experimental “control.” Any exclusion criteria should also be documented—for example, specific populations or modes of delivery of an intervention.

During critical appraisal , all studies to be included are first assessed for methodologic rigor. 3 Although there are some subtle differences, this appraisal is akin to assessing the risk of bias in reviews that ask questions related to the effectiveness of an intervention. A systematic review aims to synthesize the best evidence for clinical decision making. Assessing the validity of a study requires careful consideration of the methods used during the research and establishing whether the study can be trusted to provide a reliable and accurate account of the intervention and its outcomes. 5-8 Studies that are of low or questionable quality are generally excluded from the remainder of the review process. Exclusion of lesser-quality studies reduces the risk of error and bias in the findings of the review. 3 For the most part, critical appraisal focuses squarely on research design and the validity and hence the believability of the study's findings rather than on the quality of reporting, which depends on both writing style and presentation. 10 For example, when assessing the validity of an RCT, critical appraisal generally focuses on four types of systematic error that can occur at different stages of a study: selection bias (in considering how study participants were assigned to the treatment groups), performance bias (in considering how the intervention was provided), attrition bias (in considering participant follow-up and drop-out), and detection bias (in considering how outcomes were measured). 3

To aid the transparency and reproducibility of this process in the systematic review, standardized instruments (checklists, scales) are commonly used when asking the reviewers about the research they are reading.

Data extraction and synthesis. Once the quality of the research has been established, relevant data aligned to the predetermined outcomes of the review must be extracted for the all-important synthesis of the findings. (These steps will be addressed in the fifth paper in this series.) Data synthesized by systematic reviews are the results extracted from the individual research studies; as with critical appraisal, data extraction is often facilitated by the use of a tool or instrument that ensures that the most relevant and accurate data are collected and recorded. 3 A tool may prompt the reviewer to extract relevant citation details, details of the study participants including their number and eligibility, descriptive details of the intervention and comparator used in the study, and the all-important outcome data. Generic extraction tools for both quantitative and qualitative data are readily available. 12 The data collected from individual studies vary with each review, but they should always answer the question posed by the review. While undertaking a review, reviewers will find that data extraction can be quite difficult—often complicated by factors of the included studies such as incomplete reporting of study findings and differing ways of reporting and presenting data. When these issues arise, reviewers should attempt to contact the authors to obtain missing data, particularly for recently published research. 5

Data synthesis is a principal feature of the systematic review. 3, 6, 7, 9 There are various methods available, depending on the type of data extracted that's most appropriate to the review question. 7 An example of a systematic review addressing a question of the effectiveness of a nursing intervention is one examining nurse-led cardiac rehabilitation programs following coronary artery bypass graft surgery; the review aims to give an overall estimate of the intervention's effectiveness on patients’ health-related quality of life and hospital readmission rates. 13 Depending on the question asked, such a synthesis of the results of relevant studies also allows for exploration of similarities or inconsistencies of the treatment effect in different studies and among various settings and populations. 5 Where inconsistencies are apparent they can be analyzed further. The synthesis either provides a narrative summary of the included studies or, where possible, statistically combines data extracted from the studies. This pooling of data is termed “meta-analysis.” 14

A meta-analysis may be included in a systematic review as a practical way of evaluating many studies. Meta-analysis should ideally be undertaken only when studies are similar enough; studies should sample from similar populations, have similar objectives and aims, administer the intervention of interest in a similar fashion, and (most important) measure the same outcomes. 3 Meta-analysis is rarely appropriate when such similarities do not appear across studies. Meta-analysis requires transforming the findings of treatment effect from individual studies into a common metric and then using statistical procedures across all of the findings to determine whether there is an overall effect of the treatment or association. 8, 9, 14 The typical output from a statistical synthesis of studies is the measure or estimate of effect; the confidence interval, which indicates the precision of the estimate; and the quantification of the differences (heterogeneity) between the included studies and the statistical impact of these differences, if any, on the analysis. There are many different statistical methods by which results from individual studies can be combined during the meta-analysis. The results of the meta-analysis are commonly displayed as a forest plot, which gives the reader a visual comparison of the findings.

Owing to the limited availability of relevant trials, reviews that aim to examine the effectiveness of an intervention may resort to evidence from experimental studies other than RCTs and even from observational studies; such reviews have the potential to play a greater role in evidence-based nursing, where trials, historically, have been rare. 15 But when conducting a systematic review of studies using designs other than the RCT, a reviewer must take into account the biases inherent in those designs and make definitive recommendations about the effectiveness of a practice with caution.

Other types of evidence, including qualitative evidence and economic evidence addressing questions related to health care costs, can also be synthesized using methods established by organizations such as the Joanna Briggs Institute. 12 While the methods of synthesizing quantitative data are relatively straightforward and accepted, there are numerous methods for synthesizing qualitative research. Such reviews may appear as a meta-synthesis, a meta-aggregation, a meta-study, or a meta-ethnography 7, 16 ; the differences between these approaches will be discussed in the fifth article in this series.

A systematic review that addresses both quantitative and qualitative studies, as well as theoretical literature, is referred to as an “integrative” or “comprehensive” systematic review. 6, 15 The motivation for conducting a comprehensive review is often to provide further insight into why an intervention appears to have a benefit (or not). “Realist” reviews, another emerging form of evidence synthesis, often look to answer questions surrounding complex interventions, including how and for whom an intervention works. 7, 16 Formalized methods for these types of reviews are still being validated.

Interpretation of findings and recommendations to guide nursing practice. The conclusions of the systematic review, along with recommendations for clinical practice and implications for future research, should be based on its findings. Questions to ask when considering the recommendations of a systematic review include the following: Has a clear and accurate summary of findings been provided? Have specific directives for further research been proposed? Are the recommendations, both for practice and future research, supported by the data presented? (Such issues will be explored in the sixth and last paper in this series.)

Reviewers must consider the quality of the studies when arriving at recommendations based on the results of those studies. For example, if the best available evidence was of low quality or only observational studies were available to answer a question of effectiveness, results based on this evidence must be interpreted with caution.

Nurses are increasingly expected to make evidence-based decisions in their practice, and nursing researchers are increasingly driven to develop advanced methods of evidence synthesis. Systematic reviews aim to summarize the best available evidence using rigorous and transparent methods. We've provided a brief introduction to the steps taken in conducting a systematic review; the remaining papers in this series will explore each step in greater detail, addressing the synthesis of both quantitative and qualitative evidence.

• + Favorites
• View in Gallery

Readers Of this Article Also Read

Evidence-based practice: step by step: the seven steps of evidence-based..., systematic reviews: constructing a search strategy and searching for evidence, evidence-based practice, step by step: asking the clinical question: a key step ..., developing the review question and inclusion criteria, jbi's systematic reviews: data extraction and synthesis.

Systematic Reviews

• Systematic Review Support

What is a Systematic Review?

How long does a systematic review take, is a systematic review right for you.

• Guidelines and Protocols

Systematic reviews attempt to identify, appraise, and synthesize all evidence that meets pre-specified eligibility criteria to answer a specific research question. Systematic reviews require:

• A team of at least 2-3 researchers
• A registered protocol
• Methodology conducted in accordance to established guidelines
• Reproducible and transparent search strategies of multiple relevant databases
• Two phases of screening (i.e., Title/Abstract and Full Text) conducted by at least 2-3 researchers (3 or more researchers are recommended to help resolve disputes)
• Critical appraisal of all included studies
• Reporting in adherence with PRISMA guidelines

With five team members, an average systematic review takes approximately 62-67 weeks ( Borah et al., 2016 ):

"Review Methodologies Decision Tree" by Cornell University Library is licensed under CC BY 4.0.

• << Previous: Systematic Review Support
• Next: Guidelines and Protocols >>
• Last Updated: Aug 13, 2024 2:00 PM
• URL: https://guides.library.sc.edu/systematic-reviews

Center for Impact Sciences

Overviews in education research: a systematic review and analysis.

by esteltenpohl | Aug 15, 2024 | Access to Education , Core Components Research

The purpose of this study is to describe the prevalence and current state of overviews of education research and to provide further guidance for conducting overviews and advance the evolution of overview methods.

https://www.researchgate.net/publication/295090680_Overviews_in_Education_Research_A_Systematic_Review_and_Analysis

Recent Posts

• Core Components Approaches to Building Evidence of Program Effectiveness
• Advancing the Use of Core Components of Effective Programs: Suggestions for Researchers Publishing Evaluation Results
• Identifying and selecting the common elements of evidence based interventions: a distillation and matching model
• Understanding the Common Elements of Evidence-Based Practice: Misconceptions and Clinical Examples
• Improving Social Competence Programs for Children and Youth

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• BMC Health Serv Res

Effectiveness of registered nurses on patient outcomes in primary care: a systematic review

Julia lukewich.

1 Faculty of Nursing, Memorial University, 300 Prince Phillip Drive, St. John’s, NL A1B 3V Canada

Ruth Martin-Misener

2 School of Nursing, Dalhousie University, 5869 University Ave. St, Halifax, NS B3H 4R2 Canada

Allison A. Norful

3 School of Nursing, Columbia University, 630 West 168th Street, New York, NY 10032 USA

Marie-Eve Poitras

4 Département de Médecine de Famille Et Médecine d’urgence, Université de Sherbrooke, 2500 Boulevard de l’Université, Sherbrooke, QC J1K 2R1 Canada

Denise Bryant-Lukosius

5 School of Nursing, McMaster University, 1280 Main St W, Hamilton, ON L8S 4L8 Canada

Shabnam Asghari

6 Department of Family Medicine, Memorial University, 300 Prince Phillip Drive, St. John’s, NL A1B 3V6 Canada

Emily Gard Marshall

7 Department of Family Medicine Primary Care Research Unit, Dalhousie University, 1465 Brenton Street, Suite 402, Halifax, NS B3J 3T4 Canada

Maria Mathews

8 Department of Family Medicine, Schulich School of Medicine & Dentistry, University of Western, 1151 Richmond Street, OntarioLondon, ON N6A 5C1 Canada

Michelle Swab

9 Health Sciences Library, Faculty of Medicine, Memorial University, 300 Prince Phillip Drive, St. John’s, NL A1B 3V6 Canada

Joan Tranmer

10 School of Nursing, Queen’s University, 92 Barrie Street, Kingston, ON K7L 3N6 Canada

Associated Data

All data generated or analysed during this study are included in this published article.

Globally, registered nurses (RNs) are increasingly working in primary care interdisciplinary teams. Although existing literature provides some information about the contributions of RNs towards outcomes of care, further evidence on RN workforce contributions, specifically towards patient-level outcomes, is needed. This study synthesized evidence regarding the effectiveness of RNs on patient outcomes in primary care.

A systematic review was conducted in accordance with Joanna Briggs Institute methodology. A comprehensive search of databases (CINAHL, MEDLINE Complete, PsycINFO, Embase) was performed using applicable subject headings and keywords. Additional literature was identified through grey literature searches (ProQuest Dissertations and Theses, MedNar, Google Scholar, websites, reference lists of included articles). Quantitative studies measuring the effectiveness of a RN-led intervention (i.e., any care/activity performed by a primary care RN) that reported related outcomes were included. Articles were screened independently by two researchers and assessed for bias using the Integrated Quality Criteria for Review of Multiple Study Designs tool. A narrative synthesis was undertaken due to the heterogeneity in study designs, RN-led interventions, and outcome measures across included studies.

Forty-six patient outcomes were identified across 23 studies. Outcomes were categorized in accordance with the PaRIS Conceptual Framework (patient-reported experience measures, patient-reported outcome measures, health behaviours) and an additional category added by the research team (biomarkers). Primary care RN-led interventions resulted in improvements within each outcome category, specifically with respect to weight loss, pelvic floor muscle strength and endurance, blood pressure and glycemic control, exercise self-efficacy, social activity, improved diet and physical activity levels, and reduced tobacco use. Patients reported high levels of satisfaction with RN-led care.

Conclusions

This review provides evidence regarding the effectiveness of RNs on patient outcomes in primary care, specifically with respect to satisfaction, enablement, quality of life, self-efficacy, and improvements in health behaviours. Ongoing evaluation that accounts for primary care RNs’ unique scope of practice and emphasizes the patient experience is necessary to optimize the delivery of patient-centered primary care.

Protocol registration ID

PROSPERO: International Prospective Register of Systematic Reviews. 2018. ID=CRD42 018090767 .

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-022-07866-x.

Primary care is the foundation of a highly functioning health care system and provides comprehensive, patient-centered care that considers the needs and experiences of the individual patient, their families, and the well-being of the broader community [ 1 ]. Primary care providers are the first contact and principal point of continuing care for patients within the health care system, and coordinate other specialist care and services that patients may need [ 1 , 2 ]. The delivery of primary care occurs across varied settings but is most frequently provided in a clinic and, increasingly, by interprofessional teams that may consist of family physicians, registered nurses (RNs), nurse practitioners, pharmacists, and other health professionals. In primary care settings, RNs function as generalists and provide a broad range of patient services across the lifespan, including preventative screening, health education and promotion, chronic disease prevention and management, and acute episodic care [ 3 – 6 ]. Specifically, family physicians and RNs represent a key collaborative relationship within these teams, contributing to strengthened primary care delivery and improvements in the comprehensiveness, efficiency, and value of care for patients [ 7 – 9 ]. Internationally, nurses are increasingly embedded in primary care settings and are recognized as the most prominent non-physician contributor to primary care teams, although the scope and speed of implementation in this area differs across countries [ 10 , 11 ]. Primary care nursing in Australia is the fastest growing employment sector, with 63% of general practices employing a primary care nurse (and 82% of this group representing RNs) [ 12 , 13 ]. The World Health Organization’s report [ 14 ] on the state of the world’s nursing workforce emphasizes the need to strengthen the integration of RNs into primary care, as well as the need for further research to evaluate their impact. Global workforce data are unavailable given the variability in scope of practice and role terminology, and the lack of available information across countries. A recent review of the international literature identified that titles used to refer to RNs in primary care vary across countries [ 15 ]. For instance, titles for this role in Canada are ‘family practice nurse’ and ‘primary care nurse’, whereas in Australia, the United Kingdom, and Netherlands this title is known as ‘general practice nurse’ [ 15 ]. For the purpose of this manuscript, ‘primary care RN’ will be used throughout.

Most research in this area to date has focused on describing the roles and activities of primary care RNs. A systematic review conducted by Norful et al. [ 5 ] synthesized 18 studies from eight countries related to primary care RNs and identified assessment, monitoring, and follow-up of patients with chronic diseases as fundamental roles of the primary care RN. In contrast, there have been a number of reviews conducted on the effectiveness of nurse practitioners in primary care [ 16 – 18 ]. It is imperative that primary care RNs also begin to demonstrate their contributions to patient care within this setting. Research examining RN effectiveness has primarily been conducted within the acute care and long-term care settings and focused on staffing, role enactment, and work environment. Within these settings, there is substantial evidence demonstrating the positive effects of RN staffing on improving care and reducing adverse outcomes for hospitalized patients [ 24 , 25 ].

Furthermore, select countries including Australia, Canada, New Zealand, and the United Kingdom have developed national standards of practice or competencies to define the scope and depth of practice for primary care RNs [ 4 , 19 – 23 ]. National competencies for primary care RNs were recently published in Canada [ 7 ]. These competencies articulate the unique scope of practice and contributions to patient care for primary care RNs across six overarching domains, namely, (1) Professionalism, (2) Clinical Practice, (3) Communication, (4) Collaboration and Partnership, (5) Quality Assurance, Evaluation and Research, and (6) Leadership.

Theoretical foundation

Determining effectiveness normally requires an examination of an intervention (e.g., primary care nursing) on a particular outcome. Incorporation of the patient perspective offers a more complete understanding of the challenges patients face within our healthcare system, especially those with long-term chronic diseases. Measuring the patient experience, which is a strong predictor of quality and value of care, should be done systematically [ 26 ]. The Organization for Economic Cooperation and Development (OECD) Patient Reported Indicator Surveys (PaRIS) Conceptual Framework was developed through a comprehensive process involving extensive international collaborations and provides a roadmap and survey tools (i.e., patient and provider questionnaires) to focus the evaluation of health care interventions on patient-reported metrics [ 27 ]. This framework provides a fuller evaluation of performance by complimenting other metrics (e.g., system/cost outcomes), while also focusing attention on the needs of the patient. The main domains of the framework include: patient reported experience measures (PREMs), patient reported outcome measures (PROMs), and health behaviours (e.g., physical activity, diet, tobacco use, alcohol use). Within primary care, the PaRIS Framework can serve as a guide for routine collection of these outcomes to facilitate quality improvement and patient-centered care [ 27 ]. A growing body of research in this area has adapted the use of this model to serve as an organizational and methodological framework. For example, multiple studies have used this framework as a method of investigating the suitability and feasibility of questionnaire and survey instruments when addressing patient perspectives [ 28 , 29 ] or in the evaluation of health-related quality of life measures from the patient’s perspective [ 30 ]. A recently published systematic review that explored the opportunities and challenges of routine collection of PREMs and PROMs data for melanoma care within primary care settings found that these measures can address important care gaps and facilitate research and assessment [ 31 ]. Similarly, a study employing qualitative methods found that the use of patient-reported measures by practitioners enhanced patients’ ability to self-manage, communicate, engage, and reflect during consultations [ 32 ]. A recent environmental scan of the PROMs landscape was conducted within Canada and internationally, indicating a lack of standardized programs for routine collection and reporting of patient outcomes. Consequently, the need for enhanced PROMs information has been identified as an area of high priority [ 33 ].

Although existing literature provides some information about the contributions of primary care RNs towards outcomes of care, a systematic review synthesizing the effectiveness of the primary care RN workforce is needed. Prior to beginning the study, the Cochrane Database of Systematic Reviews, the Joanna Briggs Institute (JBI) Library of Systematic Reviews, and the Prospective Register of Systematic Reviews (PROSPERO) were searched and no existing registered protocols or previous systematic reviews on this topic were identified. Evaluating PREMs, PROMs, and health behaviours, as well as other patient-level outcomes, is necessary to accurately demonstrate the contribution of primary care RNs, hold them accountable for their care, and generate evidence to inform decisions and policies that impact their implementation and optimization [ 34 , 35 ]. Therefore, the purpose of this systematic review is to summarize evidence examining primary care RNs’ impact on patient outcomes, including physiologic changes (via biomarkers), PREMs, PROMs, and health behaviours.

A systematic review of effectiveness was conducted using JBI Systematic Review methodology [ 36 ] and findings were reported in accordance with the 2009 (and where possible, the 2021) Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) framework [ 37 , 38 ]. Covidence software was used to manage and organize the literature [ 39 ] and enable a team approach for study and data review. The protocol for this systematic review is registered on PROSPERO (registration ID CRD42018090767). This paper presents a summary of findings from studies that report on patient outcomes, including biomarkers, PREMs, PROMs, and health behaviours. A full description of the methods and findings from studies that measured care delivery and system outcomes are reported in the companion paper “Effectiveness of Registered Nurses on System Outcomes in Primary Care: A Systematic Review” [ 40 ].

Search strategy

The search strategy aimed to include both published and unpublished literature. Following a limited search in CINAHL and Medline that identified optimal search terms, two members of the research team performed a comprehensive search of relevant electronic databases (see Supplementary File 1). Grey literature was identified using ProQuest Dissertations and Theses, MedNar, Google Scholar, the websites of relevant nursing organizations (e.g., International Council of Nurses, Community Health Nurses of Canada), and reference lists of included articles. There were no location or publication date restrictions on search criteria. Studies published in any year up to and including the date of article retrieval (January, 2022) were considered. Ongoing searches for grey literature included studies with publication dates up to January, 2022.

Inclusion criteria

Studies considered for inclusion reported on any quantitative study published in English with outcomes that directly measured, or were related to, an intervention attributable to a primary care RN. Only studies focused on RNs or equivalent (e.g., practice nurse, general nurse) [ 15 ] were included; if the RN designation was unclear or could not be determined based on the region of publication, the study was excluded. Studies that involved primary care RNs who underwent considerable advanced/focused training or those that exclusively examined structural variables were excluded. Full details regarding inclusion criteria are published in the companion paper [ 40 ].

Reviewers included two study authors (DR and JL) and two trained research assistants (AR and OP). All identified titles and abstracts were screened independently by two reviewers for potential study eligibility. Two reviewers independently screened full-text articles for relevance, applying pre-established eligibility criteria. Any disagreements were resolved through discussion, or by a third reviewer.

Risk of bias

The risk of bias and quality of each study was assessed using the Integrated Quality Criteria for Review of Multiple Study Designs  (ICROMS) tool (see scoring matrix located in Supplementary File 2) [ 41 ]. All full-text articles that met eligibility criteria were appraised for quality by two independent reviewers. All studies that met inclusion/exclusion criteria also met the minimum ICROMS score to be included in the review.

Data extraction and synthesis

All eligible full-text studies underwent data extraction using a tool pre-designed and tested by the research team and based on the Cochrane Public Health Group Data Extraction Template [ 42 ]. Data extracted from the articles included: country and year of publication, study aim and design, description of primary care setting, patient sample sizes and demographics, details of study intervention and primary care RN involvement/role, outcome measures used to evaluate these interventions, and study results. To address the broad range of terms and descriptors used across included studies (e.g., traditional care, standard care, basic support, care delivered by anyone other than a primary care RN) and to provide clarity in the presentation of our results, we refer to all control groups as “usual care” or the “comparator group”. Outcomes were grouped in accordance with the OECD PaRIS Conceptual Framework Classification [ 27 ] into one of three categories defined by this model (i.e., PREMS, PROMs, health behaviours), and an additional category added by the research team (i.e., biomarkers) (see Table ​ Table1). 1 ). Biomarkers consist of outcomes related to changes in patient health status as measured by clinical assessment (e.g., hemoglobin A1c [HbA1c] values, blood pressure, body weight). PREMs are defined as patient experience indicators related to health care access, autonomy in care, and overall satisfaction with care received, and are often assessed through self-report questionnaires or population-based surveys [ 27 ]. These outcomes can be summarized further based on patient experiences surrounding access (e.g., first point of contact), comprehensiveness of care, self-management support, trust, and overall perceived quality of care. PROMs are described as outcomes relating to a patient’s self-reported physical, mental, and social health status and can be categorized as either generic or condition-specific and applied to a broad patient population [ 27 ]. Outcomes identified on this level can be further categorized into functional status (e.g., disability, physical, mental, social function), symptoms, and health-related quality of life. The remaining outcomes were categorized according to the health behaviors classification, which includes lifestyle behaviors and actions that can contribute to a patient’s overall health status (e.g., physical activity, smoking status, dietary intake) [ 27 ]. Due to the diversity of included designs, interventions, and outcomes across studies, narrative synthesis was used to present study findings.

Classification of patient outcomes measured in each study based on the OECD PaRIS Conceptual Framework [ 27 ] 

Figure  1 presents a PRISMA diagram outlining the results of the literature search.

PRISMA Diagram of Literature Search. *This paper reports on studies that measured patient outcomes. Findings from studies that measured care delivery and system outcomes are reported in the companion paper "Effectiveness of registered nurses on system outcomes in primary care: a systematic review" [ 40 ]

Study characteristics

Of the 29 articles included in the final review, 23 reported on patient outcomes (included in the present analysis) [ 40 ]. Table ​ Table2 presents 2  presents a detailed summary of the study characteristics for each of these articles. Studies were published between the years 1996–2021 and conducted in the United Kingdom ( n  = 9), United States ( n  = 6), Australia ( n  = 5), and New Zealand ( n  = 3). Study designs included randomized controlled trials ( n  = 9), observational ( n  = 8) (e.g., survey, secondary data analysis), cohort ( n  = 1), non-controlled ( n  = 2) and controlled before-after ( n  = 1), and two studies with mixed-methods designs that combined a non-controlled before-after with a non-randomized controlled trial ( n  = 1) or with an observational design ( n  = 1). Sample sizes ranged from 81–2850 patients. Quality scores, as assessed by the ICROMS tool, varied between studies. Four studies were scored at the minimum threshold for their study design [ 46 , 56 , 61 , 65 ], six studies scored 1–2 points above threshold [ 44 , 45 , 48 , 49 , 53 , 57 ], and thirteen studies exceeded the minimum cut-off score by 3 or more points [ 43 , 47 , 50 – 52 , 54 , 55 , 58 – 60 , 62 – 64 ].

Literature review table of study characteristics ( n  = 23)

*Mixed methods study consisting of multiple designs; separate ICROMS quality appraisal scores were generated for each study type; RN registered nurse, PN practice nurse, MD medical doctor, BMI body mass index,  FTE full-time equivalent, HCA health care assistant, GP general practitioner, CPS clinical pharmacy specialist, NP nurse practitioner, NOS not otherwise specified, SSRI selective serotonin reuptake inhibitor; LPN licensed practical nurse, PFMT pelvic floor muscle training

Overview of RN interventions

The nature of interventions examined in this review differed across studies. The most common interventions were related to chronic disease prevention and management, specifically, case management or targeted chronic disease management care (e.g., diabetes, obesity, hypertension, hypocholesteremia) ( n  = 7) [ 43 – 49 ] and primary and secondary preventative care for patients at risk of chronic disease (e.g. prediabetes, coronary heart disease) ( n  = 3) [ 50 – 52 ]. Other studies examined primary care RN-delivered smoking cessation support ( n  = 4) [ 53 – 56 ], back pain education and management [ 57 ], pelvic floor muscle training [ 58 ], consultations aimed at increasing patient physical activity levels [ 59 , 60 ], and a telecare program for patients with diagnosed depression [ 61 ]. Four studies examined the impact of RN care in general (at an organizational-level); three of which focused on consultations with patients in general practice [ 62 – 64 ] and another which examined the impacts of a nurse-operated telephone consultation/triage service [ 65 ].

In thirteen studies, primary care RNs carried out the intervention independently without the support of other staff/providers [ 45 , 46 , 49 , 51 , 53 – 55 , 57 – 60 , 62 , 63 ], and in 10 studies, they carried out the intervention interdependently, in collaboration with health care providers (e.g., physicians, clinical pharmacy specialists [CPS], dieticians) or members of the research team (e.g., trial nurse facilitator) [ 43 , 44 , 47 , 48 , 50 , 52 , 56 , 61 , 64 , 65 ]. Three of these 10 studies involved evaluating RNs at the general practice-level and therefore are assumed to be evaluating an interdependent role involving support of other health care providers [ 47 , 61 , 64 ]. The presence and type of comparator group also differed across study designs. Specifically, five of the included studies compared a nurse-led intervention to the same intervention led by other health care providers [ 46 , 52 , 54 , 55 , 58 ]. Other studies compared nurse-led interventions with that of ‘usual care’ not associated with nurse involvement ( n  = 4) [ 43 , 50 , 57 , 60 ], or with ‘usual care’ that was associated with reduced or alternative levels of nurse involvement ( n  = 5) [ 44 , 49 , 53 , 56 , 59 ]. The remaining studies examined the effectiveness of a primary care RN-delivered intervention on specific outcomes of care using an observational or before-after design ( n  = 5) [ 48 , 51 , 61 , 62 , 65 ], or did not contain a specific intervention, but rather, examined the impact of varying roles and practice characteristics of the primary care RN in general practice ( n  = 4) [ 45 , 47 , 63 , 64 ].

Overview of outcomes

A total of 46 patient outcomes were identified across included studies (Table ​ (Table3). 3 ). Physiologic disease control outcomes, which were measured via biomarkers, included quality of care for diabetes (e.g., HbA1c, fasting blood glucose) [ 43 , 44 , 50 , 51 ], obesity (e.g., body mass index [BMI], waist circumference) [ 44 , 47 , 50 , 51 , 59 , 60 ], pelvic floor strength and endurance [ 58 ], hypercholesterolemia (e.g., total cholesterol) [ 49 ], and hypertension (e.g., blood pressure) [ 43 , 44 , 48 , 50 , 51 ]. Patient experience outcomes identified under the PREMs category included patient satisfaction with access to care (RN versus physician as first point of contact) [ 64 ], quality of self-management support (e.g., smoking cessation counseling, chronic disease services) [ 56 , 62 ], comfort/trust with primary care RN roles [ 45 ], and overall satisfaction or perceived quality of care with provider consultations, treatment, or advice/support received [ 45 , 51 , 55 , 57 , 63 , 65 ]. Patient reported outcomes identified within the PROMs category consisted of physical and social functional status [ 43 , 57 ], level of disability (e.g., activity levels, bed rest, work loss) [ 57 , 61 ], changes in self-reported anxiety, depression, or pain [ 59 – 61 ], adverse health events (e.g., falls, fractures, severe hypoglycemia) [ 43 , 59 , 60 ], and health-related qualify of life (e.g., physical activity, social activity) [ 43 , 46 , 51 , 52 , 60 ]. Lastly, outcomes grouped under the health behaviors classification included reduction and/or cessation of tobacco use [ 51 , 53 – 56 ], changes to physical activity (e.g., level of aerobic exercise, daily step count) [ 51 , 57 , 59 , 60 ], and improvements in dietary intake [ 49 ].

Literature Review Table – Description of Patient Outcomes and Study Results

HbA1c hemoglobin A1c, RN registered nurse, BMI body mass index, CHD coronary heart disease, PN practice nurse,  OR odds ratio, CI confidence interval, CPS clinical pharmacy specialist, BP blood pressure,  ES effect size, cmH 2 0 centimetres of water pressure, GP general practitioner, HCA health care assistant, MBS Medicare Benefits Schedule, MVPA moderate to vigorous physical activity, EPAT Eating Pattern Assessment Tool

Physiologic disease control via biomarkers

Ten studies measured clinical patient outcomes when comparing interventions that involved primary care RNs to that of usual care or an intervention delivered by a comparator group. Clinical biomarkers included those for diabetes (HbA1c, fasting blood glucose), obesity (BMI, total fat mass), hypertension (blood pressure), and cardiovascular risk (total cholesterol). Of the ten studies, four examined diabetic control. After one year, Aubert et al [ 43 ]. reported significant differences in HbA1c values; patients in the primary care RN case management group had a larger mean reduction (-1.7 percentage points) over 12 months in comparison to usual care (-0.6 percentage points) (difference -1.1, 95% CI: -1.62 to 0.58; p  < 0.001). Additionally, patients in the intervention group had a greater decrease in fasting blood glucose than the usual care group (-48.3 mg/dL versus -14.5 mg/dL; difference -33.8, 95% CI: -56.12 to 11.48;  p  = 0.003). Bellary et al [ 44 ]. found a small but non-significant reduction in HbA1c among their patient sample after two years. One additional study that conducted a retrospective data analysis of clinical outcome data from patients attending an independently RN-led primary care clinic, did not detect significant changes in HbA1c between initial intake at baseline and follow-up visits at various intervals (reported as 3 months to “several years” depending on the individual) [ 51 ].

Seven studies examined obesity-related outcomes such as BMI, weight, and total fat mass. In their adjusted regression models, Karnon et al [ 47 ]. reported that high level involvement of primary care RNs in the provision of obesity-related clinical activities (in comparison to low level involvement) yielded significantly larger mean reductions in BMI (mean difference -1.10, 95% CI: -0.45 to -1.76; p  = 0.001) after one year, however, there were no significant improvements in terms of the proportion of patients losing weight (mean difference 0.09, 95% CI: -0.07 to 0.25;  p  = 0.259). Coppell et al [ 50 ]. found a significant weight reduction (-1.3 kg) in the primary care RN-led prediabetes intervention arm compared to usual care (gained 0.8 kg) (2.2 kg difference; p  < 0.001). Mean BMI and waist circumference also decreased in the intervention arm compared to an increase in the control group, however, these differences were not significant. Likewise, a third study reported that fat mass was slightly reduced at 12 months, but differences between the intervention and control groups were equivalent when the primary care RN group was compared to both postal intervention ( p  = 0.54) and usual care ( p  = 0.30) [ 60 ]. There were no significant reductions in BMI or waist circumference in the remaining four studies [ 43 , 44 , 51 , 59 ].

Five studies investigated the impact of enhanced nurse involvement in primary care delivery on blood pressure. Bellary et al [ 44 ]. reported significant differences between groups in diastolic blood pressure (-1.91, 95% CI: -2.88 to -0.94 mm Hg; p  = 0.0001) and mean arterial pressure (1.36, 95% CI: -2.49 to -0.23 mm Hg; p  = 0.018), favoring the intervention (additional time spent with a primary care RN). In a second study conducted by O’Neill et al. in which the RN independently assessed blood pressure and collaborated with either a CPS or physician in hypertension case management, [ 48 ] there was a greater decrease in systolic blood pressure in patients who received care from CPS and primary care RN teams (-14 mm Hg) compared to patients receiving care from physician-directed primary care RNs (-10 mm Hg) ( p  = 0.04), however, there were no significant changes in diastolic blood pressure between groups. The remaining three studies found no significant changes in blood pressure when comparing a primary care RN-led intervention to that of usual care [ 43 , 50 ] or from initial baseline to follow-up [ 51 ].

Total cholesterol was measured in four studies. Pine et al  [ 49 ]. reported that the mean total cholesterol level decreased by 0.29 mmol/L (11.2 mg/dL) (4.3%) from the initial physician visit to the first primary care RN visit. Following five counseling sessions by a primary care RN, the mean total cholesterol levels of all patients decreased (-0.14 mmol/L; p  = 0.4). However, during the follow-up comparison study, there were no significant differences in total cholesterol improvement between the nurse-counseling intervention group and the comparison patients, and total cholesterol levels in both groups improved significantly ( p  = 0.002). The remaining three studies reported equivalent results in regards to total cholesterol reduction [ 43 , 44 , 50 ].

Patient experience outcomes via PREMS

Nine articles reported on patient experience outcomes via PREMs: overall perceived quality of care [ 45 , 51 , 55 , 63 , 65 ], self-management support [ 56 , 62 ], access (first point of primary care contact) [ 64 ], comprehensiveness [ 57 ], and trust [ 45 ].

In regard to overall perceived quality of care, Halcomb and colleagues [ 45 ] found that Australian patients were very satisfied and comfortable with chronic disease care delivered by a primary care RN. This was particularly true for patients with diabetes who reported being almost three times more comfortable (38% versus 14%, p  = 0.016) with their encounter than patients who consulted for other chronic health conditions. A similar study in New Zealand also revealed high satisfaction with primary care RN-delivered services overall, with increased satisfaction associated with an increased number of visits (i.e., those who had more than four previous visits to the primary care RN) after controlling for demographic factors [ 63 ]. Longer consultation time with a primary care RN resulted in higher patient satisfaction (OR = 2.50, 95% CI: 1.43 to 4.35; p  < 0.01) and patient enablement (OR = 2.55, 95% CI: 1.45 to 4.50; p < 0.01) than shorter consultation time [ 62 ]. Moreover, patients who attended practices where primary care RNs worked with broad scopes of practice and high levels of autonomy were more satisfied (OR = 1.76, 95% CI: 1.09 to 2.82; p  = 0.04) and more enabled (OR = 2.56, 95% CI: 1.40 to 4.68; p  < 0.01) than patients who attended practices where nurses worked with more limited scopes of practice and lower levels of autonomy [ 62 ].

Patients also reported improved health, better understanding of disease diagnosis, medication, and treatment plan, and more motivation for self-management as a result of primary care RN-led lifestyle clinics focused on diabetes, smoking cessation, women’s health, cardiovascular risk, respiratory/asthma, and diet/nutrition [ 51 ]. Furthermore, patients reported positive experiences with primary care RN-led telephone consultations for acute illness [ 65 ], back pain education [ 57 ], and smoking cessation support [ 55 , 56 ]. For instance, Cherkin et al [ 57 ]. reported higher satisfaction ( p  < 0.0) and higher perceived knowledge ( p  < 0.001) for patients who received a primary care RN-led educational intervention for back pain than those in the usual care group. Nearly all patients (98%, n  = 385) in an Australian study [ 56 ] that examined smoking cessation behavioral support from a primary care RN rated the support provided as helpful (19%) or very helpful (79%) and indicated that they may have been more successful with smoking cessation if they had been able to have more sessions with the RN. With respect to access to care, a study by Caldow et al [ 64 ]. found that patients expressed satisfaction and preference with primary care RN versus physician consultations for minor illness as first point of contact if this resulted in a reduced waiting time, suggesting that patients would be accepting of an expanded nursing role in primary care.

Patient reported outcomes via PROMs

Patient reported outcome measures via PROMs were examined across eight studies and included health-related quality of life [ 43 , 46 , 51 , 52 , 60 ], symptoms [ 59 – 61 ], self-efficacy [ 60 ], and functional status [ 57 ].

Health-related quality of life, as measured through patient self-report, was assessed in five studies [ 43 , 46 , 51 , 52 , 60 ]. In a 12-month randomized controlled trial conducted by Aubert et al., [ 43 ] a primary care RN-led case management model of adult diabetes care was compared with that of usual care in a primary care setting. Health-related quality of life was assessed by a validated questionnaire developed by the Centers for Disease Control and Prevention for the Behavioral Risk Factor Surveillance System to assess patient perception of health status across four domains. The results demonstrated an improved perception of health status in both groups, with patients in the intervention group more than twice as likely to report improvement in health status score (mean change = 0.47) as those in the usual care group (mean change = 0.20) ( difference= 0.27; 95% CI: -0.03 to 0.57; p  = 0.02). In contrast, the other four studies [ 46 , 51 , 52 , 60 ] examining health-related quality of life did not report significant differences in regards to these outcomes, including two cluster randomized controlled trials [ 46 ,  52 , 60 ]. One study assessed the effectiveness of three different methods of secondary prevention care of coronary heart disease (recall to a primary care RN; recall to a physician; audit and feedback) [ 52 ], while the other compared the efficacy of a primary care RN-supported physical activity intervention to that of usual care [ 60 ]. Both studies reported equivalent scores between groups on all dimensions of patient self-reported quality of life measurements. Lastly, an observational study by Marshall et al [ 51 ]. used patient satisfaction surveys to assess perceptions of Nurse-Led Healthy Lifestyle Clinics (NLHLC) in New Zealand. Using scores from the Dartmouth Primary Care Cooperative (COOP) Information charts [ 66 ], it was noted that there were no statistically significant differences in the COOP dimensions related to self-perceived quality of life from first clinic visit to last clinic visit. However, significant improvements were noted in relation to the COOP variables related to patient-perceived social activity (mean difference = -0.20; p  = 0.049), change in health (mean difference = -0.42; p  = 0.001), and overall health (mean difference = -0.21; p  = 0.025).

Patient self-reported symptoms were measured in three studies [ 59 – 61 ] and included outcomes related to both mental and physical health (anxiety, depression, pain), as well as the occurrence of adverse health events (injuries, fractures, cardiovascular events, deaths, and deterioration of any pre-existing health problems). In an observational study of a nurse telecare intervention for adults with depression in the United States, Pearson et al [ 61 ]. found a significant improvement in mean scores on the SF-12 Mental Functioning Scale between baseline (mean = 29.9) and 6-months post-intervention (mean = 48.2) ( p  < 0.0001). During the same time interval, significant differences were noted on the Hamilton Depression Rating scale (14.6 to 6.5; p  < 0.001), as well as the mean scores on the Work Limitations Questionnaire (70.4 to 87.2; p  < 0.001), which both represent an improvement in functioning. Paired t-test results for the difference in mean scores on all three instruments were statistically significant ( p  = 0.0001) and the majority of patients (63%) experienced at least a 50% reduction in the Hamilton Depression Rating score at 6-months. The remaining studies to examine self-reported symptoms as an outcome were two randomized controlled studies that measured the effects of a primary care RN-delivered intervention on patient physical activity [ 59 , 60 ]. Both studies assessed changes to patient self-reported levels of depression, anxiety, and pain and incidents of adverse health events. The results of both studies reported no statistically significant between-group differences in mean scores of either symptom at 3- or 12-months post-intervention. Additionally, while total number of adverse events did not differ between groups for either study, Harris et al [ 60 ]. found a significant reduction in cardiovascular events among the intervention group over the 12 month period ( p  = 0.04).

Patient self-efficacy was examined in a three-arm cluster randomized controlled study conducted by Harris et al [ 60 ]., in which patient-reported levels of exercise self-efficacy were examined at 3- and 12-months following a physical activity intervention. Exercise self-efficacy in this study was characterized by a patient’s willingness to set goals, create action plans, engage in self-monitoring, and seek out social support, and are directly related to long-term physical activity adherence. Findings indicated that exercise self-efficacy was significantly increased in both intervention groups at 3-months for postal group (pedometer delivered by mail) versus control (Effect Size [ES] = 1.1, 95% CI: 0.2 to 2.0; p  = 0.01), primary care RN group versus control (ES = 2.3, 95% CI: 1.4 to 3.2; p  < 0.001), and primary care RN group versus postal group (ES = 1.2, 95% CI: 0.3 to 2.1; p  = 0.01). For primary care RN group versus control group, the difference remained significant at the 12-month follow-up (ES = 1.2, 95% CI: 0.3 to 2.2; p  = 0.01), but not for the postal group versus control ( p = 0.2) or the primary care RN group versus postal group ( p  = 0.22).

The sole study to evaluate functional status was a randomized controlled trial conducted by Cherkin et al [ 57 ]. comparing usual care, usual care plus an educational booklet, and usual care plus an educational session with a primary care RN and an educational booklet to improve outcomes of low back pain in primary care. None of the interventions had a statistically significant effect on functional status, including days of limited activity, bed rest, or work loss resulting from back pain one week after the intervention or at any subsequent follow-up.

Health behaviors

Of all the studies included ( n  = 23), nine considered health behavior outcomes. Among the studies examining the impact of a primary care RN-led intervention on health behaviors, it was found that tobacco use was the most documented health behavior ( n  = 5) [ 51 , 53 – 56 ]. Tobacco use was examined by looking at both abstinence from smoking, as well as daily reductions or changes to smoking behaviour, and this was measured at multiple follow-up periods throughout the duration of the intervention. All five studies demonstrated positive changes in smoking-related health behaviors following either an independent [ 51 , 53 – 55 ] or interdependent [ 56 ] primary care RN intervention. For example, Byers et al [ 54 ]. compared a primary care RN-led intervention with a physician-led intervention to support smoking cessation. The results show that support provided by the primary care RN was equivalent to that provided by the comparator group (29.1% versus 18.2% quit rate, respectively; p  = 0.077) in supporting the patient to quit smoking. Marshall et al [ 51 ]. looked at primary care RN-led healthy habits lifestyle clinics for patients with or at risk of chronic disease within targeted populations with known health inequalities. Following the intervention, 94% of patients reported having a better understanding of their diagnosis, medication and treatement plan, and an increase in motivation to self-manage their health needs. Other studies have examined the impact of nursing interventions on patient-reported levels of physical activity [ 57 , 59 , 60 ]. Only one study considered adherence to healthy eating as a health outcome following a RN-led intervention in primary care [ 49 ]. In this study, Pine and colleagues evaluated the effect of a nursing intervention to support cholesterol lowering for patients diagnosed with hypercholesterolemia. To do this, primary care RNs provided a total of five counseling visits focused on nutritional education and physical activity (1-month after referral, and at 3-, 5-, 7-, and 12-months) to 82 patients with total cholesterol higher than 6.21 mmol/l. Intervention patients were already following a diet consistent with the program at baseline, however, the mean score for Section 1 of the Eating Pattern Assessment Tool (questions related to foods with serum cholesterol-raising potential) improved from 23.4 at the first visit to 20.4 at the final visit ( p  < 0.001).

This systematic review presents a comprehensive synthesis of literature examining the impact of primary care RNs on patient outcomes. The findings suggest that outcomes resulting from care provided by primary care RNs are comparable and complementary to care provided by other primary care providers, specifically with respect to chronic disease prevention and management, smoking cessation, and wellness counseling. This review supports that primary care RNs deliver appropriate and high-quality patient care. There was a high level of patient satisfaction reported regarding experiences with RN-led care. Patients appear to be comfortable with RNs providing primary care services and taking on expanded roles in primary care. This is consistent with findings from other studies that have examined patient satisfaction and comfort with RN roles in primary care practices across multiple countries [ 67 – 70 ]. Our findings are aligned with existing evidence that has linked patient experiences of care to the level of autonomy and scope of practice of the RN in the clinical setting. A recently published study from Canada evaluated patient experiences in primary care organizations and determined that patient-reported experience was significantly enhanced in clinics in which RNs systematically followed patients and used their scope of practice to their full potential [ 71 ].

Patient experience is a strong indicator of patient-perceived quality of care and fundamental to achieving desired patient outcomes for a range of physical and mental health domains [ 72 – 76 ]. The recently developed PaRIS Framework served as a valuable tool for organizing patient-reported outcomes, however, it did not capture all patient outcomes identified within the studies included in this review. For example, clinical biomarkers, such as HbA1c and fasting blood glucose (used as a measure of diabetes care quality), were not considered in the PaRIS Framework and therefore added by the study authors during the analysis phase as an additional patient outcome category. Similarly, studies evaluating RN interventions in primary care did not consider many components of the PaRIS Framework, such as delivery system design (e.g., clinic remuneration, remote consultations), individual and sociodemographic factors (e.g., patient demographic characteristics), and health and health care capabilities, and only measured select components from the patient-reported experiences of care, health behaviours, and patient-reported outcome domains. For instance, many articles did not provide information regarding RN characteristics, such as level of education, years of experience, or specific roles/tasks that they performed in-clinic prior to the intervention that might have impacted outcomes observed. In addition, conceptual definitions of outcomes within included studies may vary and not align with meanings as defined within the PaRIS Framework. A taxonomy for RN outcomes may be useful and should be considered in future revisions and applications of the OECD PaRIS Framework.

The included studies evaluated a variety of primary care RN interventions but did not capture all roles that encompass their broad scope of practice. Commonly offered services by primary care RNs that have yet to be comprehensively evaluated include prenatal and well-baby care, therapeutic interventions (e.g., wound care, treatment of infections), preventative care (e.g., immunizations, health promotion and education) and care coordination (e.g., nursing surveillance, system navigation). Although self-management supports (e.g., smoking cessation, physical activity, diabetes, nutrition, pain management, healthy lifestyle promotion, chronic disease prevention) were examined in a few studies [ 43 , 44 , 49 – 51 , 53 – 57 , 59 , 60 ], further evaluation of self-management and behaviour support interventions (included within the PaRIS Framework) offered by RNs in primary care is needed. Moreover, recently developed competencies identify that, in addition to clinical practice activities centered around patient care provision, primary care RNs engage in a wide range of non-clinical roles, such as leadership, research, and interprofessional collaboration. These non-clinical domains of practice for primary care RNs require further understanding and evaluation.

There is a general apprehension among some medical practitioners that if RNs assume more responsibilities or enhanced roles within primary care settings, high-quality care and patient safety will be compromised [ 77 , 78 ]. Findings from this study show that patient care is equivalent to that of “usual care” and in many cases, produced better patient outcomes when the intervention was provided by a primary care RN. This aligns with literature in the acute care and long-term care settings [ 24 , 25 ]. The findings from our review call into question concerns that RN-provided care increases risk or reduces quality of care and equally, lends support towards the efficacy of primary care RN care provision on improvements to patient outcomes. Additionally, primary care RNs are required to practice within their legislated and regulated scope of practice, regardless of the practice setting or types of clinical roles performed [ 79 , 80 ].

Generally, there are methodological challenges associated with examining the contributions of a specific health care provider within the context of a team [ 81 – 83 ]. As the focus of practice and research moves towards interdisciplinary teams, it is increasingly difficult to isolate and evaluate the impact of primary care RN interventions. In addition, the roles of primary care RNs and team compositions vary across practice settings. Although this review exclusively examined studies in which an intervention was delivered by a RN, many studies were excluded because of unclear terminology surrounding the nursing designation/role (i.e., unable to discern whether nurse in study was a RN or equivalent). Furthermore, the review included only nine randomized controlled trials, which provide the strongest level of evidence, that specifically compared RN-led interventions to care delivered by other health care professionals and/or usual care [ 43 , 52 , 53 , 57 – 60 , 84 , 85 ]. Comparator groups in studies varied considerably, impacting the ability to make comparisons across studies and limiting the generalizability of findings from this study. Despite these challenges, this review provides preliminary evidence on patient outcomes used to evaluate a variety of different RN interventions in a multidimensional health care environment. The findings from this study, coupled with an existing framework (e.g., OECD PaRIS Framework) serve as a tool to map roles and activities to outcomes and guide future evaluation of primary care RNs. Overall, as the presence of RNs in primary care increases globally, further evaluation research implementing control/comparison groups into study design and controlling for confounding factors (e.g., nurse characteristics) is needed to strengthen the evidence related to the effectiveness of RNs in primary care.

Strengths and limitations

This review provides preliminary evidence regarding the effectiveness of RNs on patient outcomes in primary care. Traditional means of measuring the effectiveness of care provision in the healthcare sector have focused mainly on the use of clinical data. A strength of this systematic review is its patient-oriented approach that assesses health outcomes from the patient perspective [ 27 , 86 ]. Additional strengths of this systematic review include the application of a comprehensive search strategy and use of the PRISMA checklist in the screening process. However, despite utilizing a comprehensive search strategy, it is possible that not all relevant studies were retrieved and included in this review. Furthermore, although we conducted an appraisal using an established quality assessment tool (i.e., ICROMS), this tool presented certain challenges. For instance, although a strength of this tool is that it offered criteria to assist with the process of assigning quality scores, there is a degree of subjectivity involved in the appraisal process. In addition, the minimum cut-off scores varied across study designs and therefore, made comparisons of the quality between different types of studies difficult. Similarly, the score itself is difficult to interpret without an understanding of the tool and design matrix (limitations of articles are summarized in Supplementary Table 3). The ICROMS tool was also not designed to specifically appraise mixed methods or observational designs. The lack of consistent and available data regarding terminology used to describe RNs, or equivalent nursing titles, across countries limited the ability to include studies published in certain regions. Only studies published in the English language were included, which may limit generalizability to certain countries. High-quality research employing robust study designs (e.g., randomized controlled trials) need to be conducted to further understand the impact of RNs on patient outcomes in primary care.

Primary care RNs can provide patient care that is comparable and complementary to that of other primary care providers, specifically with respect to patient satisfaction, enablement, self-reported quality of life, self-efficacy, and improvements in health behaviours. This review provides preliminary evidence regarding the effectiveness of RNs on patient outcomes in primary care. Findings are applicable to researchers and other stakeholders engaged in primary care reform and can inform further integration and optimization of this role, as well as contribute to future research.

Acknowledgements

We acknowledge the following research assistants for the contributions to the systematic review screening, appraisal and data extraction process: Richard Buote, Ashley Joyce, Olivia Parsons, and Arifur Rahman.

Abbreviations

Authors' contributions

JL conceived of the larger project, obtained grant funding, supervised data collection and screening, interpreted and synthesized results, and drafted and revised the manuscript; DBL, RMM, AN, MEP interpreted results and drafted and revised the manuscript; MS carried out the initial search strategy, assisted with adherence to PRISMA guidelines, interpreted results, and revised the manuscript; SA, EGM, MM, JT interpreted results and revised the manuscript; DR screened, appraised and extracted data, and assisted with interpretation of results and the drafting/revising of the manuscript. All authors approved the final draft.

This research was supported by funding received from Memorial University and the Department of Health & Community Services, Government of Newfoundland & Labrador.

Availability of data and materials

Declarations.

Not applicable.

The authors declare that they have no competing interests.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Ohio State nav bar

The Ohio State University

• BuckeyeLink
• Find People
• Search Ohio State

Newly-published study tests tool to gauge nurses’ confidence in evidence-based practice implementation strategies

Tool aims to address gaps and help incorporate research to improve patient care

A study published this week in the Western Journal of Nursing Research highlights preliminary data for validity of a new tool that helps measure the confidence of nurses in implementation strategies to accelerate the adoption of evidence-based practice in the name of improved patient care and safety.

The study was done by three faculty and staff members from the Helene Fuld Health Trust National Institute for Evidence-based Practice (EBP) in Nursing and Healthcare at The Ohio State University College of Nursing: Sharon Tucker, PhD, APRN-CNS, NC-BC, EBP-C, FNAP, FAAN; Inga Zadvinskis, PhD, RN; and Linda Connor , PhD, RN, CPN. They developed what the study calls the Implementation Self-efficacy for EBP Scale (ISE4EBP) to measure nurses’ confidence in using EBP implementation strategies. As the publication’s abstract states, “Clinicians may use the scale to identify areas for building implementation confidence to accelerate the uptake of evidence to improve patient care.”

The two-phase study used the tool with staff nurses and nurse leaders and tested the validity and reliability of the responses, as well as how the tool measured confidence. The highest mean scores related to working with leadership, education, use of communication strategies such as fliers or visual aids and mentorship. Low scores related to items such as equipment, obtaining human resources, administrative issues and data dashboards/auditing. According to the authors, “As nurse ratings of organizational readiness scores improved, nurses’ self-efficacy for implementation improved, consistent with our expectations based on the conceptual framework linking self-efficacy and work context.”

“This study gives us a new baseline to work from as we work to fast-track the implementation of evidence-based practice within healthcare organizations,” said Tucker, who also serves as both Grayce Sills Endowed Professor in psychiatric-mental health nursing and director of the College of Nursing’s DNP nurse executive track. “Research tells us that it takes upwards of 17 years to translate research into routine practice, often due to stalls in the implementation and change process. We also know that evidence-based practice improves the quality and safety of patient care, enhances population health, saves money and increases job satisfaction for clinicians. Anything we can do to help nurses and other clinicians build confidence in strategies for implementation can cut down the timeframe to incorporate EBP into practice.”

The Fuld team is currently analyzing data from parallel studies (one with more than 500 nurses with similar findings) and planning future work to develop a guide for interpreting scores for high or low self-efficacy, as well as test interventions designed to build confidence in strategies and enable sustainable practice changes and patient outcomes.

Related Articles

There can be no doubt about the effect the COVID-19 pandemic had on the profession of nursing. A study recently released by The National Council of State Boards of Nursing (NCSBN) points to factors including burnout and stress that have made lasting impacts on the profession.

This month’s Transformative Solutions in Healthcare piece features the full text of an interview with Arlene Bierman, MD, MS, director of the Center for Evidence of Practice Improvement and the federal Agency for Healthcare Research and Quality (AHRQ) . The interview was conducted on March 28, 2022 by Bernadette Melnyk, PhD, APRN-CNP, FAANP, FNAP, FAAN, vice president for health promotion, university chief wellness officer and dean of the College of Nursing.  

Ohio State’s College of Nursing dean and Chief Wellness Officer recognized for a lifetime of impactful science

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Carrying out systematic literature reviews: an introduction

Affiliation.

• 1 Lecturer in Health Data Science, School of Health Sciences, University of Manchester, Manchester.
• PMID: 31393770
• DOI: 10.12968/bjon.2019.28.15.1008

Systematic reviews provide a synthesis of evidence for a specific topic of interest, summarising the results of multiple studies to aid in clinical decisions and resource allocation. They remain among the best forms of evidence, and reduce the bias inherent in other methods. A solid understanding of the systematic review process can be of benefit to nurses that carry out such reviews, and for those who make decisions based on them. An overview of the main steps involved in carrying out a systematic review is presented, including some of the common tools and frameworks utilised in this area. This should provide a good starting point for those that are considering embarking on such work, and to aid readers of such reviews in their understanding of the main review components, in order to appraise the quality of a review that may be used to inform subsequent clinical decision making.

Keywords: Health care education; Health care roles; Nursing education; Nursing evaluation research; Nursing research.

PubMed Disclaimer

Similar articles

• Student and educator experiences of maternal-child simulation-based learning: a systematic review of qualitative evidence protocol. MacKinnon K, Marcellus L, Rivers J, Gordon C, Ryan M, Butcher D. MacKinnon K, et al. JBI Database System Rev Implement Rep. 2015 Jan;13(1):14-26. doi: 10.11124/jbisrir-2015-1694. JBI Database System Rev Implement Rep. 2015. PMID: 26447004
• The effectiveness of internet-based e-learning on clinician behavior and patient outcomes: a systematic review protocol. Sinclair P, Kable A, Levett-Jones T. Sinclair P, et al. JBI Database System Rev Implement Rep. 2015 Jan;13(1):52-64. doi: 10.11124/jbisrir-2015-1919. JBI Database System Rev Implement Rep. 2015. PMID: 26447007
• The effectiveness of public health nursing: the problems and solutions in carrying out a review of systematic reviews. Elliott L, Crombie IK, Irvine L, Cantrell J, Taylor J. Elliott L, et al. J Adv Nurs. 2004 Jan;45(2):117-25. doi: 10.1046/j.1365-2648.2003.02873.x. J Adv Nurs. 2004. PMID: 14705995
• Experiences of registered nurses as managers and leaders in residential aged care facilities: a systematic review. Dwyer D. Dwyer D. Int J Evid Based Healthc. 2011 Dec;9(4):388-402. doi: 10.1111/j.1744-1609.2011.00239.x. Int J Evid Based Healthc. 2011. PMID: 22093388 Review.
• Health professionals' experience of teamwork education in acute hospital settings: a systematic review of qualitative literature. Eddy K, Jordan Z, Stephenson M. Eddy K, et al. JBI Database System Rev Implement Rep. 2016 Apr;14(4):96-137. doi: 10.11124/JBISRIR-2016-1843. JBI Database System Rev Implement Rep. 2016. PMID: 27532314 Review.
• Design and Implementation Factors for Performance Measurement in Non-profit Organizations: A Literature Review. Treinta FT, Moura LF, Almeida Prado Cestari JM, Pinheiro de Lima E, Deschamps F, Gouvea da Costa SE, Van Aken EM, Munik J, Leite LR. Treinta FT, et al. Front Psychol. 2020 Aug 7;11:1799. doi: 10.3389/fpsyg.2020.01799. eCollection 2020. Front Psychol. 2020. PMID: 32903643 Free PMC article.

Publication types

• Search in MeSH

LinkOut - more resources

Full text sources.

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.