The Impact of Immersive Virtual Reality on Learning, Post-Hoc: a Cautionary Tale

Danielle Eve Schneider
Northern Illinois University

The use of Immersive Virtual Reality (VR) and Immersive Virtual Reality Learning Environments (IVRLEs) in an educational setting is connected to learning outcomes, motivation and assessment and has profound implications toward the future of educational technology. Even though IVRLEs are not new to the classroom, they have become increasingly more complex and immersive in nature and are beginning to drive research foci to look not only at cognitive needs, but instruction as well.

VR presents the user with a multisensory-stimulating interactive experience. It does this by creating a false sense of "being there," called presence. This allows individuals to dismiss what they see as a false-reality and engage, or "immerse" into their virtual environment (Bailey, Bailenson, & Casasanto, 2016; Lee, Sergueeva, Catangui, & Kandaurova, 2017; Slater & Steed, 2000). This active, situational learning experience allows users to gather information, solve complex problems and actively engage their imagination. Additionally, through the immersive interactions of a head-mounted display (HMD) connected to the realistic computer environment, the user is able to interact and envelope oneself in the virtual world. In the simplest of terms, this replication of the three-dimensional space, allows the user a visual-spatial perspective which creates complete sensory engagement (Bailey, 2016; Ehinger et al., 2014; Pavone et al., 2016; Riva et al., 2007).

Theoretical Foundations

When input is received by the brain through the VR experience, it does not have an immediate, tangible, physical context, and therefore it defers to stored memory. This is an important point when thinking about how the brain is processing the VR. While one may assert that "knowledge is created and accumulated through direct experiences" (Lee, 2017), the Embodied Cognition Theory takes it one step further and recognizes that memories are indexed through multiple senses and are made of more than just their visual cue (Black, 2010; Lakoff & Johnson 1992; Thompson, 2007; Varela, Thompson, & Rosch 1993). The Immersive Cognition Theory (ICT) builds upon these and provides a more holistic explanation into the pathways by which we process information in the VR environment (Ladendorf, Schneider, & Xie, 2019). When individuals engage in the VR environment, there is direct access to embodied, long-term memories (LTM) as they interact with the visual register and bypass the working memory (WM) (Atkinson & Shiffron, 1968; Foxe & Snyder, 2011; Ladendorf et. al., 2019, Lenggenhager, Halje, & Blanke, 2011), thus placing less strain on the working memory (WM). Research has shown that new learning experiences benefit from this established sense of presence as it channels direct access to the LTM, as a cognitive correlate from previously indexed life experiences (Pan, Bingham, & Bingham, 2017; Pavone, 2016; Ladendorf et al., 2019). Looking at individuals beyond the initial experience in a classroom or other virtual experience, the Immersive Cognition Model identifies an overload channel, whereby levels of engagement can be impacted due to an overwhelming influx of visual input. Ultimately, because the LTMs are accessed and evoke more than just the visual sense, a sensory overflow which can have lasting impacts on the user is created (Ladendorf et al., 2019).

Much research has identified that we must provide digitally engaging and information-rich content to ensure our learners are engaged within the system (Rupp et al., 2016). The challenge is to identify how our population's engagement levels toward learning in other modalities transition as they exit the IVRLE with the implications being potentially detrimental to outside learning experiences. Understanding the fundamental construct that engagement drives learning, three subtypes of engagement were investigated through this survey; affective, cognitive, and behavioral, as well as disengagement. Each has its own unique traits yet retains the independent influence to impact an individual's learning potential (see Table 1) (Boykin & Noguera, 2011).

Utilizing the aforementioned theoretical foundations, this study sought to identify how student engagement was impacted due to this potential overloading effect. Was there a difference, post-hoc, in students' engagement levels when comparing the treatment group to the non-treatment group? Additionally, what types of engagement were most influenced when learning within the IVRLE, post-hoc?

Methods

Sample

The data for this study were collected from forty-two 7th grade public school students from a suburb of a large U.S. city. Their ages ranged from 11-13. Groupings were based on convenience sampling, and the treatment group (n=23) and control group (n=19) were divided based on the student's existing classroom assignment. Students were of mixed ethnic descent and socioeconomic background. Groupings included all students in the existing class roster, including English Language Learners (ELL) and students with Individual Education Plans (IEPs) and other special accommodations, as offered in 504 plans. Students were not given any tracking information connecting to their identity.

Participation was completely voluntary, and individuals did not receive any extra-credit nor compensation for their collaboration. All students participated in the initial survey to provide a baseline data point two weeks before the onset of the experiment. Students were also not informed whether they would be placed in the treatment or control group.

Experimental Design and Materials

The student virtual environment system was run through Apple iPhone and Google Cardboard Viewers with the standard classroom setup following the Google Expedition guidelines ("Expeditions resources," 2018). The students utilized the cardboard viewers with the Apple iPhone connected through an internal wireless network to a teacher-guided VR program running Google Expeditions: Grand Canyon and Geology, directed by an Apple iPad. The non-technology infused lesson included identical, printed visuals taken from the Google Expeditions and the language guiding the instruction was comparable.

The classroom engagement inventory survey instrument was taken from Wang, Bergin, and Bergin (2014). The questions were divided into 4 sections: Behavioral Engagement (BE) Cognitive Engagement (CE), Disengagement (DE), and Affective Engagement (AE). Within each engagement category, there were four to seven questions to measure each individual set of engagement.

The identical survey was provided to each participant and each student used the same platform and computer to complete their surveys. All questions were prefaced with "In this class" and students responded using the Likert Rating Scale of 1(strongly agree) to 5 (strongly disagree). This survey was completed in an online format. The participant data went into a spreadsheet, which was coded and exported into IBM SPSS v25 for statistical analysis.

Procedure

Two weeks prior to the experiment, all students and parents were informed of the experiment and assigned a randomly generated identification number to be used when completing the engagement surveys which would track their responses anonymously. Students were unaware of whether they would be part of the treatment or control group at this point.

The treatment group was provided with a twenty-five-minute science lesson through VR, then they were given the engagement survey. Immediately following the survey, the students began a second twenty-five minute non-technology enriched science lesson. Following this lesson, they took the second identical survey. The control group was provided with identical content, including the non-technology enriched lesson that aligned with the VR lesson; they then completed the same survey as the treatment group. Immediately following, they participated in a second identical non-technology enriched science lesson. When complete, the students responded to the second identical survey. This experimental design is indicated in Figure 1.

Statistical analysis

The analysis of the student responses was completed through a repeated measure ANOVA to determine the efficacy of the virtual environment affecting engagement post-hoc as compared to the non-treatment group. The initial survey students completed was used as the co-variate. Additionally, Cronbach's Alpha was run to ensure consistency across responses and confirmative factor analysis by Varimax rotation method with Kaiser normalization was completed to validate the grouping of the questions within the survey tool. 

Results

Cronbach's Alpha yielded mixed results (see Table 2). AE showed strong reliability with survey 1= 0.870 and survey 2= 0.920. CE yielded reliable results, though not as robust, with survey 1= 0.752 and survey 2= 0.776. Consequently, both BE and DE were unable to affirm positive results through Cronbach's Alpha for both survey 1 and 2, as shown in Table 2. Even when specific items were removed, the value was not raised significantly to support reliability across the responses.

The Confirmative Factor Analysis (CFA) data aligned with the Cronbach's alpha results and identified strong reliability in the AE survey question subset CE as well. However, the confirmative factor analysis results for CE were not as robust compared to those of AE. Lastly, the BE and DE data did not fit the measurement model within CFA. A repeated measures ANOVA was completed for each sample set of engagement questions with the treatment and control group with the two time points of data collection (survey 1 and survey 2). In order to normalize the data, the pre-survey was applied as the co-variate to the repeated measures ANOVA (Table 3).

The data showed that there was a significant difference between the means of VR exposure on students' AE F_1,39=10.263; p=.033 and BE F_1,39=7.045; p= .011, versus the control group. However, the data suggests that there was no significant difference between the control group and the treatment group on CE with a F_1,39=.428; p=.517 and Disengagement (DE) F_1,39=.428; p=.750, as shown in Table 3 and Figure 2.

Lastly, the baseline characteristics of the treatment and control group were evaluated to ensure there were no differences academically between the two groups. This was done by attaining the students' Science-MAP (Measure of Academic Performance) test scores and confirmed with an independent samples T-test for equal means that yielded no significant difference between the two with p > 0.05.

Missing Data

No outliers were removed; however, some students were assigned an ID number and did not complete a given time point survey (N=9). The data from these individuals was determined to be unusable and their entire data set was removed from the study. This happened for a number of reasons. Some students were absent on the day of the treatment. Other students did not follow directions to complete the survey. Some students did not remember to write down their ID number, which was randomly generated on the first day of the study and there was no way to retrieve it and attach it back to the individual, and others experienced physical limitations (discussed below) that made them unable to complete the activity. These data had no impact on the outcome of this study.

Discussion

This study explored the impact of the immersive VR learning environment on student engagement post-hoc. Additionally, it analyzed which types of engagement were affected by exposure through immersive VR learning. Looking at the individual engagement components, the results strongly support the hypothesis that affective engagement is affected post-hoc when participating in an immersive VR learning environment. The data provides a statistically significant correlation which was affirmed through confirmatory factor analysis.

This engagement component is a reasonable assertion when thinking about the format of the learning environment. When students are participating in an IVRLE, there is a significant amount of positive engagement that is stimulating and exciting, and when there is a return to the traditional classroom, their "feelings" are brought back down to earth, and they often drop below that excitement point where they feel a sense of disappointment. This resulted in lower values of positive emotions reported toward the learning environment.

The research additionally provided data to support a similar hypothesis for behavioral engagement. This would relate the on-task, connected behavior to their learning. Again, in this instance, when the students are highly engaged in the interface of an IVRLE, they feel the sense of excitement and specifically connection to their learning, once this environment has been removed, they are left with a sense of longing which results in the decrease in mean levels of engagement, in the treatment population. This is what provides a clear and relevant perception of disassociation to learning post-hoc, and this is what is most troubling about the use of this technology in an educational setting. When the tool is removed from the learning environment, students are left with a longing for the tool, and a disassociation for learning. While a concept may prove interesting for a learner, the tool or IVRLE may remain the focus or even distraction, and therefore, the overall engagement with learning post-hoc decreases.

Although this study was unable to support a relationship between cognitive engagement and learning post-hoc, the results may not be entirely incongruent. Students' perceived value of learning is often subjective and while individuals may not be interested in the topic, lack of interest does not infer the lack of value (Fredricks, Blumenfeld & Paris, 2004). Additionally, lack of cognitive learning does not always infer lack of focused during a learning session. What was identified during the IVRLE sessions was that many of our students were paying attention but not focusing. One student responded, "I was so excited and wanted to look around, that I couldn't entirely focus on what you were telling me to do." As the students become engaged with the IVRLE sensory input, we see that additional verbal cues can act as overload, as identified through the Immersive Cognition Theory (Ladendorf et. al., 2019).

Additionally, the failure to support the hypothesis with the disengagement set was mostly due to questions that were not aligned to an IVRLE. Additionally, the discrepancies identified from the Cronbach's Alpha and CFA will need to be investigated further with a survey tool that is more VR specific.

Research Limitations

Examining the results more closely, the questionnaire posed significant challenges to this study. For example, within the grouping of behavioral engagement, one of the questions asked about class discussion, while the other four questions did not connect on the same level through the lens of a VR lesson. This became problematic within the survey tool in general, as was noted when looking at the CFA. Students were also informally interviewed for feedback about the survey itself. One student shared, "At first I was totally shocked that you asked if I let mind wander in class and wondered if anyone would tell you the truth if they did, but now I totally get it, because you do let your mind go when you're in VR!" Another student said, "I was confused during the VR lesson about the question that asked if we search for information - how are we supposed to search while we're using the Google Cardboard viewers?" These two statements were just a few examples of responses directed toward the survey that identified respondent's confusion and potential inconsistencies within the data. Fundamentally, these statements exposed significant flaws within this tool, as it does not apply to a classroom that relies on students "letting go" and experiencing a virtual world (Rupp, 2016).

Another challenge for this experiment was the unexpected onset of visually induced motion sickness (VIMS) by a few participants. While the underlying cause of VIMS is not known, it occurs when the brain anticipates movement due to the visual cues it receives from the virtual viewer, since it is not experiencing the corresponding motor cues. This sensation of 'self-motion' is often significant enough to create the disconnect called 'VIMS' (Kasharvarz & Hecht, 2015; Lee, 2017; Bonato, Bubka, Palmisano, Phillip & Moreno, 2008; Gramann et al., 2010; Jiwon, Mingyu, & Jinmo, 2017; Chen, Chao, Chen, Wang, & Tang, 2015; Young Youn, Hyun Ju, Eun Nam, Hee Dong & Hyun Taek, 2005; Howarth & Hodder, 2008).

Conclusion and Future Directions

In order to understand and determine the learning benefits of the IVRLE, one must look to the complete outcomes of the experience. VR creates a phenomenological parallel when one enters. Learning within VR is fundamentally different than through other means, as the environment is reality, but the experience is not. When our students are receiving and processing multiple brain stimuli through the IVRLE, they are experiencing the phenomena of Embodied Cognition, but how that experience truly affects us remains a mystery (Ladendorf et al., 2019).

Exposure to VR should not be viewed as a discrete experience. Its impact has a broad reach. Using this study to identify the effects of engagement post-hoc, we can support educational technology with a more holistic view of the essential components regarding the integration of VR learning and education. Additionally, looking at the resulting cognitive processes that our students experience while in the IVRLE is an important component of the learning environment which should not be overlooked (Ladendorf et al., 2019).

The significance of these findings has opened the door to investigate more profound secondary effects of VR, which will continue to present challenges to our fast-paced, technology-rich learning environments. While there has been much research in the VR field and its benefits to support all aspects of experiential learning, including support for individuals who are physically or mentally handicapped (Arvind et al., 2014), there have not been investigations to identify the impact on these individuals of their experience with the VR environment. While the rush of adrenaline is the obvious lasting effect from a rollercoaster, what is the lasting effect from a 'high density,' visual learning experience for a significant amount of time?

Research continues to focus on what is happening while 'in' the VR environment, or what types of learning can occur. However, there is not the same fervor to identify the effects to learning after individuals exit VR, identifying the effects post-hoc, as they continue their instructional day in the traditional learning environment. More importantly, a point which cannot be underscored enough is that this technology is being pushed into schools and developing minds without the due diligence or foresight of identifying any potential outcomes and risks.

Knowing that the use of VR in education continues to advance, this study has also identified the need to create a VR engagement survey tool that accurately and clearly identifies specific levels of engagement within the VR environment. While claims of increased student learning and retention through virtual environments are anecdotal or do not use a standard measure (Fowler, 2015), educators must be able to look to the research for validated claims if we expect to use VR in the future of education. If we don't, we run the risk of creating virtual classrooms that look and feel like the real ones but miss the mark at creating "pedagogically new and innovative learning environments" (Fowler, 2015, p. 416). Moreover, if we are building and implementing a standard measuring tool around a group of disconnected or archaic constructs, research will be unable to provide an accurate report to the value and efficacy of the IVRLE.

In conclusion, this research has provided the first insight into the effects the IVRLE post-hoc. Affective and behavioral engagement identified a statistical significance, suggesting an impact beyond the experience of VR learning. When individuals are exposed to an IVRLE, the long term effects are not yet completely known and require further research. This study has laid groundwork for further research to follow. Additionally, this research identified a substantial need for a standardized engagement tool of measure for the IVRLE, in order for data to be normalized and applicable to all end-users.

Author Note

Danielle Eve Schneider
Educational Technology, Research and Assessment Department
Northern Illinois University

Correspondence concerning this article should be addressed to
Danielle Schneider
737 Deerpath Dr.
Deerfield, IL 60015
Email: sciencewithmaldo@gmail.com