Development of the undergraduate simulation experience is needed to continue developing aspiring clinicians’ ability to assess and manage unwell patients. However, its heavy resource demand and ‘Generation Z’s’ desire for technologically enhanced learning means novel simulation modalities must now be considered. Mixed reality (MR) provides a platform to deliver such simulation; however, it is yet to be studied in this context. We conducted an observational cohort study with the aim to assess ift MR simulation improves students’ perceived ability to assess and manage unwell patients.
We recruited 32 undergraduate medical students and asked them to rate their ability to assess and manage acutely unwell patients, out of 10. An MR simulation was then delivered using the Microsoft HoloLens and HoloScenario software produced by GigXR. Students then rated their assessment and management ability again, alongside their experience of the system and its usability. Data analysis used paired t-tests to assess for significant differences.
By attending the simulation, students showed a significant improvement in perceived ability to assess (p < 0.001) and manage (p < 0.001) acutely unwell patients. Mean assessment scores improved by 1.09/10 (95% confidence interval [CI 0.67, 1.52]) with 89% of students feeling more confident. Mean management scores improved by 1.63/10 (95% CI [1.15, 2.10]) with 84% of students reporting increased confidence levels. When considering usability, 69% of students did not find the MR system easy to use, with 75% of students having technical issues and 38% experiencing side effects. Overall, 88% of students believed the experience was beneficial to their learning.
MR shows promise in its ability to deliver simulation training and improve students’ perceived ability to assess and manage unwell patients. Advances in software availability and content are required for integration into undergraduate medical curricula. Further research is required to assess if these results are replicated objectively.
Simulation within medical education is the creation of an environment that allows a learner to feel as close to a real-life event as possible, behaving and reacting to situations they find themselves in without the risk of real-world repercussions [1,2]. The simulation process allows learners to develop competence in many areas including knowledge, skills, communication and teamwork through experiential learning, by developing from previous learning events and progressing to the higher levels of Bloom’s taxonomy such as application, analysis and evaluation [3,4]. The simulation process has been shown to be significantly more effective in knowledge transfer and skill acquisition than traditional didactic teaching styles, allowing students to engage in repetitive practice and gain immediate feedback [5].
However, high-quality simulation training is extremely resource consuming [6]. This becomes an even more pressing problem when appreciating the current economic and social climate, with the current societal demands to increase both the quantity and quality of medical student training [7]. A recent review identified that ‘Generation Z’ students require vibrant learning environments that combine social interaction with advanced technology for optimal engagement due to cultural and economic differences during their development [8–10]. This contrasts that of ‘millennials’ who are the subject of most medical education research, and with the natural paradigm set to shift towards having more Generation Z students, research should focus on the needs of future learners [11].
The Microsoft HoloLens is a head-mounted device (HMD) that delivers a mixed reality (MR) experience. MR is a new educational and technological advancement, which allows learners to experience the real world combined with a digital world, through the use of holograms [12]. This represents a distinct concept from virtual reality in which the user is completely removed from the real world. The use of MR has been researched in multiple modalities within medical education, including its use in the delivery of bedside teaching [13,14], clinical skills [15,16] and anatomy [17,18]. This literature has demonstrated the ability for MR to significantly contribute to learning, with the focus largely centred around the feasibility of this teaching modality, as well as demonstrating a significant improvement in the acquisition and development of clinical skills. However, its use in simulation for the management of acutely unwell patients is yet to be studied.
The development of better high-quality simulation will help shape future clinicians, but the resource demand and limited capability of simulators create a barrier to this. Currently, there is no evidence to suggest that MR simulation has utility in developing undergraduates’ ability to assess and manage acutely unwell patients. Therefore, the aim of this study was to determine if MR simulation results in a subjective improvement in students’ self-rated ability to assess and manage acutely unwell patients, with the secondary outcome measure being to better understand how students perceive an MR learning environment.
From January 2023 to March 2023, 32 medical students were recruited through convenience sampling from Leicester University at the University Hospital of Derby and Burton NHS. Participation was made available to third- and fifth-year students and there were no exclusion criteria.
A pre-session questionnaire was completed asking students to rate their ability to assess and manage acutely unwell patients. Students then attended a 2-hour teaching session in groups of three, which involved a 15-minute orientation to the Microsoft HoloLens device; and then a simulation of an acutely unwell asthmatic patient. Students worked together to progress through the simulation, which was provided via GigXR HoloScenarios software [19]. The session was led by the principal investigator who did not participate in the simulation unless the students required technical assistance. A formal simulation debrief occurred after the intervention, prior to the completion of the post-session questionnaire.
Participant demographics and previous experiences with acutely unwell patients, simulation and extended reality simulation were collected using an online questionnaire (Google Forms), as well as the students’ self-rated ability scores on a Likert scale from 1 (poor) to 10 (expert). The self-rating was then repeated after the simulation exercise, as well as seeking students’ feedback on the use of the HMDs and the GigXR software. They were also asked about problems with the session and asked to comment specifically on these issues where appropriate.
Statistical analyses were carried out using the ‘Statistical Package for Social Sciences’ software (SPSS, Version 27, 2020, Chicago, IL). The data were normally distributed. Significance testing was performed using paired t-tests. The Bonferroni correction was used to account for multiple testings with a p-value of <0.003 considered statistically significant. Student self-reported scores are marked out of a total of 10, and averages are reported as mean ± standard deviation. Self-reported score improvement was also compared across academic year, age groups and achievement of higher educational qualification, to evaluate if demographics impacted their perception of MR simulation. Although correlated, age and higher educational achievement were both evaluated to assess the impact of previous educational experience. Data regarding student experience are expressed as descriptive statistics.
All students consented to be included in the study. Ethical approval was obtained from Leicester University Medical School and Keele University School of Medicine School – Student Project Ethics Committee. Student data were collected anonymously and consent for study inclusion could be withdrawn up to the point of publication. Participation in the study was not mandatory, and students were free to leave the session at any point.
Of the 32 students participating in the study, 14 (44%) were third-year medical students and the remaining 18 (54%) were in their final year of study. There were 7 (22%) students aged 18–21, 20 (63%) students aged 22–25 and 5 (16%) students aged 26–29. There were 10 (31%) students who had attained a higher educational qualification prior to starting their medical degree.
Prior to the simulation exercise, students rated their ability to assess an acutely unwell patient as 6.2/10 ± 1.1. After the simulation, they rated their ability as 7.2/10 ± 1.1, as shown in Figure 1. There was a significant difference in students’ self-rated ability to assess an unwell patient after the simulation (t31 = 5.27, p < 0.001, d = 1.17), and self-reported scores increased by 1.09 (95% confidence interval [CI 0.67, 1.52]).
Student self-rated assessment scores before and after MR teaching session.
Students rated their ability to manage an acutely unwell patient as 5.5/10 ± 1.1. After the simulation, they rated their management ability as 7.1/10 ± 1.4, as shown in Figure 2. There was also a significant difference in students’ self-rated management ability (t31 = 7.00, p < 0.001, d = 1.31), and self-reported scores increased by 1.63 (95% CI [1.15, 2.10]). These results are shown in Table 1.
| n | Mean score before | Mean score after | Mean improvement | p-Value | Cohen’s d | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Assessment | Management | Assessment | Management | Assessment | Management | Assessment | Management | Assessment | Management | |||
| Overall | 32 | 6.2/10 ± 1.1 | 5.5/10 ± 1.1 | 7.2/10 ± 1.1 | 7.1/10 ± 1.4 | +1.09 | +1.63 | <0.001 | <0.001 | 1.17 | 1.31 | |
| Year group | Year 3 | 14 | 5.7/10 ± 0.9 | 4.9/10 ± 0.9 | 7.1/10 ± 1.2 | 6.9/10 ± 1.4 | +1.40 | +2.00 | <0.001 | <0.001 | 1.09 | 1.30 |
| Year 5 | 18 | 6.6/10 ± 1.1 | 5.9/10 ± 1.0 | 7.4/10 ± 1.0 | 7.3/10 ± 1.4 | +0.80 | +1.30 | <0.001 | <0.001 | 0.79 | 0.84 | |
| Age group | 18–21 | 7 | 5.7/10 ± 1.1 | 4.7/10 ± 1.1 | 7.1/10 ± 1.2 | 7.0/10 ± 1.0 | +1.40 | +2.30 | 0.016 | 0.001 | 1.13 | 1.13 |
| 22–25 | 20 | 6.2/10 ± 1.2 | 5.7/10 ± 1.0 | 7.2/10 ± 1.1 | 6.9/10 ± 1.3 | +0.95 | +1.20 | <0.001 | <0.001 | 0.89 | 0.95 | |
| 26–29 | 5 | 6.8/10 ± 0.4 | 6.0/10 ± 0.7 | 8.0/10 ± 0.7 | 8.4/10 ± 1.7 | +1.20 | +2.40 | 0.070 | 0.009 | n/a | 1.14 | |
| Prior educational achievement | Undergraduate | 22 | 6.2/10 ± 1.2 | 5.3/10 ± 1.1 | 7.2/10 ± 1.0 | 7.0/10 ± 1.1 | +1.00 | +1.60 | <0.001 | <0.001 | 1.00 | 1.04 |
| Higher educational qualification | 10 | 6.2/10 ± 0.9 | 5.9/10 ± 1.0 | 7.4/10 ± 1.3 | 7.4/10 ± 1.9 | +1.20 | +1.50 | 0.003 | 0.005 | 0.92 | 1.27 | |
Student self-rated assessment scores before and after MR teaching session.
The third-year group (n = 14) rated their assessment ability as 5.7/10 ± 0.9 prior to the simulation and 7.1/10 ± 1.2 afterwards, as shown in Figure 1. This showed a significant improvement (t13 = 4.91, p < 0.001, d = 1.09) and a mean improvement of 1.4 (95% CI [0.80, 2.06]). Their management ability was rated as 4.9/10 ± 0.9 prior to the simulation and 6.9/10 ± 1.4 afterwards, as shown in Figure 2. This showed a significant improvement (t13 = 5.75, p < 0.001, d = 1.30) and a mean improvement of 2.0 (95% CI [1.25, 2.75]).
The fifth-year group (n = 18) rated their assessment ability as 6.6/10 ± 1.1 prior to the simulation and 7.4/10 ± 1.0 afterwards, as shown in Figure 1. This showed a significant improvement (t17 = 4.50, p < 0.001, d = 0.79) and a mean improvement of 0.8 (95% CI [0.44, 1.22]). Their management ability was rated as 5.9/10 ± 1.0 prior to the simulation and 7.3/10 ± 1.4 afterwards, as shown in Figure 2. This showed a significant improvement (t17 = 6.73, p < 0.001, d = 0.84) and a mean improvement of 1.3 (95% CI [0.92, 1.75]). These results are shown in Table 1.
Age groups were evaluated in groups to appreciate any difference between ‘Generation Z’ (18–21 and 22–25) and ‘Millennials’ (26–29).
The 18–21 age group rated their assessment ability as 5.7/10 ± 1.1 prior to the simulation and 7.1/10 ± 1.2 afterwards, as shown in Figure 1. This did not show a significant improvement (t6 = 3.33, p = 0.016, d = 1.13) with a mean improvement of 1.4 (95% CI [0.43, 2.48]). Their management ability was rated as 4.7/10 ± 1.1 prior to the simulation and 7.0/10 ± 1.0 afterwards, as shown in Figure 2. This showed a significant improvement (t6 = 6.35, p = 0.001, d = 1.13) and a mean improvement of 2.3 (95% CI [1.41, 3.17]).
The 22–25 age group rated their assessment ability as 6.2/10 ± 1.2 prior to the simulation and 7.2/10 ± 1.1 afterwards, as shown in Figure 1. This showed a significant improvement (t19 = 4.79, p < 0.001, d = 0.89) and a mean improvement of 0.95 (95% CI [0.53, 1.37]). Their management ability was rated as 5.7/10 ± 1.0 prior to the simulation and 6.9/10 ± 1.3 afterwards, as shown in Figure 2. This showed a significant improvement (t19 = 5.64, p < 0.001, d = 0.95) and a mean improvement of 1.2 (95% CI [0.75, 1.65]).
The 26–29 age group rated their assessment ability as 6.8/10 ± 0.4 prior to the simulation and 8.0/10 ± 0.7 afterwards, as shown in Figure 1. This did not show a significant improvement (t4 = 4.50, p = 0.07), and a mean improvement of 1.2 (95% CI [0.16, 2.56]). Their management ability was rated as 6.0/10 ± 0.7 prior to the simulation and 8.4/10 ± 1.7 afterwards, as shown in Figure 2. This also showed no significant improvement (t4 = 4.71, p = 0.009, d = 1.14) with a mean improvement of 2.4 (95% CI [0.98, 3.82]). These results can be seen in Table 1.
The graduate group rated their assessment ability as 6.2/10 ± 0.9 prior to the simulation and 7.4/10 ± 1.3 afterwards, as shown in Figure 1. This did not show a significant improvement (t9 = 4.12, p = 0.003, d = 0.92) with a mean improvement of 1.2 (95% CI [0.54, 1.89]). Their management ability was rated as 5.9/10 ± 1.0 prior to the simulation and 7.4/10 ± 1.9 afterwards, as shown in Figure 2. This also did not show a significant improvement (t9 = 3.73, p = 0.005, d = 1.27) with a mean improvement of 1.5 (95% CI [0.59, 2.41]).
The undergraduate group rated their assessment ability as 6.2/10 ± 1.2 prior to the simulation and 7.2/10 ± 1.0 afterwards, as shown in Figure 1. This showed a significant improvement (t21 = 4.91, p < 0.001, d = 1.00) and a mean improvement of 1.0 (95% CI [0.60, 1.49]). Their management ability was rated as 5.3/10 ± 1.1 prior to the simulation and 7.0/10 ± 1.1 afterwards, as shown in Figure 2. This showed a significant improvement (t21 = 7.58, p < 0.001, d = 1.04) and a mean improvement of 1.6 (95% CI [1.22, 2.14]). These results can be seen in Table 1.
Only 2 of the 32 students (6%) stated they had no experience in the assessment and management of acutely unwell patients, whereas 14 students (43%) had simulation experience and 16 students (50%) had real patient experience. Regarding extended reality simulation experiences, 27 of the 32 students (84%) had no experience, 4 students (12%) had 1–3 experiences and 1 student (3%) had 8+ experiences. When considering importance, 29 (91%) believed simulation training was an important part of medical education, and 31 students (97%) believed they had further learning needs in the assessment and management of acutely unwell patients (indicated by agree or strongly agree). These results are shown in Table 2.
| Strongly agree | Agree | Neutral | Disagree | Strongly disagree | ||||
|---|---|---|---|---|---|---|---|---|
| I believe simulation training is an important part of my medical education | 59.4% | 31.3% | 9.4% | 0.0% | 0.0% | |||
| I believe I have future learning needs in the assessment and management of acutely unwell patients | 59.4% | 31.3% | 9.4% | 0.0% | 0.0% | |||
| No experience | Simulated experience only | Real patient experience | ||||||
| I have previous experience in the assessment and/or management of an acutely unwell patient | 6.3% | 43.8% | 50.0% | |||||
| No Experience | 1–3 Simulation experiences |
4–7 Simulation experiences |
8+ Simulation experiences |
|||||
| I have previous experience with simulation training within my medical education training | 3.1% | 40.6% | 28.1% | 28.1% | ||||
| I have previous experience with the use of virtual reality within my medical education training | 84.4% | 12.5% | 0.0% | 3.1% | ||||
As shown in Table 3, when considering the MR experience, 30 (94%) students stated they were able to see the patient as if they were in the room, and 16 (50%) felt they were able to interact with the patient as if they were in the room (indicated by agree or strongly agree).
| Strongly agree | Agree | Neutral | Disagree | Strongly disagree | |
|---|---|---|---|---|---|
| I was able to see the patient as if they were in the room | 53.1% | 40.6% | 0.0% | 6.3% | 0.0% |
| I was able to interact with the patient as if they were in the room |
25.0% | 25.0% | 28.1% | 18.8% | 3.1% |
| I found the HoloLens system easy to use | 3.1% | 34.4% | 21.9% | 37.5% | 0.0% |
| I found the teaching session (including initial set up) was efficiently delivered | 50.0% | 43.8% | 3.1% | 3.1% | 0.0% |
| The teaching was relevant to my learning needs | 71.9% | 25.0% | 3.1% | 0.0% | 0.0% |
| The HoloLens was beneficial to my learning in the context of this teaching session |
37.5% | 50.0% | 12.5% | 0.0% | 0.0% |
| As a result of this teaching session, I feel more confident in the assessment of acutely unwell patients | 31.3% | 46.9% | 18.8% | 0.0% | 0.0% |
| As a result of this teaching session, I feel more confident in the management of acutely unwell patients | 28.1% | 56.3% | 12.5% | 3.1% | 0.0% |
Looking at usability, 12 (31%) students found the MR system and software easy to use and 24 (75%) had technical difficulties during the session. Technical issues included issues with logging into the software (or suddenly logging out) and image calibration. When asked about side effects, 12 (38%) experienced some degree of adverse symptoms from the HMDs. Adverse effects included headache (n = 9) and dizziness (n = 3).
Overall, 28 (88%) students believed the experience was beneficial to their learning, with 25 (89%) students feeling more confident in the assessment of the acutely unwell patient and 27 (84%) students feeling more confident in their management ability. These results can be seen in Table 3.
In this observational cohort study, the delivery of a simulation through MR was evaluated in the context of assessing and managing acutely unwell patients. This was assessed through self-reported ability of undergraduate students, as well as evaluating the systems’ usability and acceptability to the students. The results demonstrate that MR-delivered teaching resulted in a significant improvement in their self-reported ability to assess and manage acutely unwell patients, as well as most students indicating that the MR system was beneficial to their learning. This is unsurprising, given the previously demonstrated benefits of the use of extended reality within medical education such as improving immersion and realism [20]; however, to the authors’ knowledge, this has not been demonstrated in undergraduates’ acute assessment and management ability until now.
This study utilized students’ self-reported scoring. Research indicates that although students, on average, tend to accurately estimate their ability, it is often subject to an overestimation bias which is more commonly seen in lower-scoring students, and may ultimately lack construct validity [21,22].
Overall, the results indicate that the students benefitted from the MR system in both perceived assessment and management scores. Interestingly, the results indicate that the third-year medical students found the MR system resulted in a greater improvement in their ability to assess and manage acutely unwell patients than their fifth-year colleagues. The more junior students can be assumed to have a lower level of baseline knowledge and have experienced less simulated and real-life experiences of unwell patients, resulting in a larger knowledge gap and more potential to learn [23]. Bloom’s taxonomy [24] describes a hierarchical model of learning that students are expected to progress through as they develop their knowledge or skills in a particular area. At the point of this learning intervention, the third-year students are likely to be at the ’understanding’ and ‘applying’ stage, learning to put their preclinical knowledge into practice. Whereas the fifth-year students would be expected to be closer to the ‘analyse’ and ‘evaluate’ levels, in which they can use their knowledge and experience to make appropriate assessment and management decisions. This potentially explains the difference in improvement, as the third-year students have more scope to improve at the time of the intervention. It is important to appreciate that this research looked at subjective measures only, the accuracy of which may naturally be variable, and further objective evidence is required to confirm this observation. Although a significant improvement in results was observed in different age groups and those with higher educational achievement, the evidence does not suggest one particular group found this more beneficial than another; however, this may be a result of small participant numbers in this initial study.
Students within the 26–29 age group did not display a significant improvement in their scores, nor did those with a previous higher educational achievement. There is likely a significant participant overlap within these groups, but ultimately indicates that those that fall into the ‘Millennials’ age group may not gain as much educational benefit from an MR simulation than those within ‘Generation Z’. This finding supports the emerging evidence that the two distinct generations may learn in significantly different ways, and as educators, we must try to reflect this in practice [11]. Furthermore, this finding demonstrates the need to fully evaluate the use of mixed reality and other extended reality platforms to support the learning of the future generation of medical students.
Interestingly, the results indicate students perceived management ability improved more than their assessment ability. No current literature has studied this phenomenon; however, it is likely linked to a lower baseline knowledge in this area, to begin with, or could simply be due to curriculum design and the sequence in which students are taught these skills. As previously discussed, it is understood that lower-scoring students are subject to overestimation bias, this may be reflected when looking at individual aspects of students’ scoring, with overestimation of their weaker areas [21].
As discussed, multiple studies demonstrate the utility of mixed realty, and the results of this study also highlight these. This study demonstrates that effective simulation can be delivered to undergraduates, without the need for multiple trained facilitators and simulated patients, as the intervention described utilized a single facilitator. It is understood that medical simulation is a costly intervention, but the true cost once equipment and materials, faculty, location and learner costs are considered is poorly quantified [6]. The prospect of an alternative and cost-saving intervention is exciting, given the huge utility simulation has demonstrated in improving educational outcomes such as patient safety, workplace efficiency, skill competence, knowledge and confidence [25]. However, it is important to appreciate that the amount of human resources required to run MR sessions is inversely proportional to the amount of available HMDs. This study required one facilitator per three students, meaning a large amount of teaching hours were required to deliver the sessions. Current MR software is capable of having higher student numbers per session; however, this study was limited by headset availability. When considering the integration of MR simulation into curricula, this balance must be carefully considered.
Simulation has been shown to be superior to traditional didactic teaching and problem-based learning, across a large proportion of the UK undergraduate curriculum including the management of acutely unwell patients, demonstrated through improved performance in formal assessment [26,27]. However, the barriers previously discussed prevent further integration and expansion of simulation into medical school curricula. The results of this study show promise in helping to overcome these barriers; however, more research is required before large-scale investment is made. Objective evidence in the use of MR with medical students is required; however, software availability and content are likely to hinder the completion of such investigations. Currently, the amount of MR software available is rapidly expanding, and it is only a matter of time before available content can fulfil the needs of undergraduate medical curricula. Qualitative research looking to assess which specific aspects of MR simulation are most effective is key to drive software developers to produce programmes that support the evidence base. Future research needs to assess if the integration of MR simulation is feasible within a medical school curriculum and crucially if MR simulation can objectively improve student performance.
Within this study, the number of participants was limited by the number of HMDs available for use; however, multiple teaching sessions were run to improve participant number. Recruitment bias was minimal as all students within the cohort were invited. Due to being a pilot study, improvement was measured subjectively and may not reflect objective improvement but provides a platform for future research.
Extended-reality (including MR) hardware and software are growing at an exponential rate. The demands of the younger generations within medical school implore the current medical education community to fully evaluate the utility of such technology, to meet their needs and further develop available educational resources. The evolving psychology and learning needs of future medical students demand that we no longer practise using outdated medical education theory. This study shows MR has a potential to meet these needs, demonstrating an improvement in self-reported assessment and management ability, as well as confidence. However, further and wider work within the medical education research community is needed to fully evaluate and appreciate the true utility of MR within medical education.
Thank you to the Technology Enhanced Learning team at NHS England for providing the hardware and software access through the extended reality hub scheme, which allowed this research to be completed.
None declared.
None declared.
None declared.
None declared.
None declared.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Twitter
Facebook
WhatsApp
LinkedIn
Tweets