The purpose of this study is to examine virtual reality (VR) learning exercises as an addition to a traditional computer-based training (CBT) course in an industrial contractor safety training organization. The Health and Safety Council (HASC) delivers an average of more than 1,500 CBT courses a day to contractors working in the energy industry, primarily focused on safety and skills training. This study is designed to determine if CBT’s could be made more effective in helping learners retain information for longer durations with the addition of VR exercises.
This study is designed to determine the effectiveness of VR exercises in relation to: (a) the impact on learners' long-term memory development of learning objectives, (b) learner acceptance of VR technology by demographic makeup of the learner and (c) identify effective VR strategies to develop a model of best practices for the learning community that the organization supports. The overarching question that guides this study is how will the use of VR technology influence the long-term memory development and memory resilience among HASC’s learners.
Research Design and Rationale
This study is designed to follow a quantitative research method using: (a) a CBT course on an aspect of industrial fieldwork, (b) VR exercises, (c) ungraded quizzes to reinforce learning objectives, (d) a survey to capture learner demographic data, and (e) a chatbot style quiz delivered to the learner one week after the completion of the CBT course.
The data acquired from the student responses to the chatbot-style quiz and survey will be used to gain insight into the impact VR technology has on learner retention of procedures and course information. The research questions that were developed to help guide this study were designed to gain insight into learner retention, learner engagement, and learner motivation.
Research Questions
Research Question 1: What is the relationship between learner retention of the material in the long term using CBT with VR exercises compared to learners without the opportunity to take VR exercises?
Research Question 2: What is the relationship between learner motivation when using CBT’s with VR exercises?
Research Question 3: What relationships exist among learner retention and motivation in relation to VR exercises?
3a. What relationships exist among learner retention and motivation in relation to learner age?
3b. What relationships exist among learner retention and motivation in relation to learner experience in industry?
3c. What relationships exist among learner retention and motivation in relation to learner gender?
3d. What relationships exist among learner retention and motivation in relation to learner level of education?
3e. What relationships exist among learner retention and motivation in relation to learner race or ethnicity?
Null Hypothesis
The null hypothesis used for this experiment is that learners who take a CBT class with VR exercises will not have higher scores on the chatbot quiz as the learners who take a CBT course without VR exercises.
Sample Design
The sample population for this study includes contractors from four different industrial service contractors visiting HASC in Pasadena, Texas who are assigned to take either confined space awareness course (19CS) or confined space awareness course with VR exercises (19CSVR) by their contracting organizations. The course codes, 19CS and 19CSVR, are unique to HASC and are used to help identify the courses by the researcher and the contracting organization when learners register for classes. Learners who take 19CS during this time period will be assigned to the control group, while learners who take 19CSVR will be assigned to the experimental group. The courses are the same price and take approximately the same amount of time. Additionally, the courses contain the same material presented in the same manner and style, except for the addition of VR exercises in 19CSVR.
Because HASC is a contractor training organization, the researchers working on this project had no control over who was assigned to take either course nor in influencing the contracting organization to select or use the two different courses. This registration process helped to influence an experimental design that focused on a random sample from learners who registered for and completed either 19CS or 19CSVR during one month from the selected contractor organizations.
Usually learners taking either 19CS or 19CSVR come from a variety of different contracting organizations who support a variety of different industrial processes, therefore they could have a variety of different experiences in the procedures trained in 19CS and 19CSVR. Although the learner population for this experiment all will have previous experience with the confined space awareness concepts provided within both 19CS and 19CSVR course, the degree of that experience might differ; from brief conceptual, theoretical understanding, to deep experiential understanding. Some of this variability comes from the type of work the contracting organization focuses on. To limit this possibility, the sample population will consist of learners who take the course in July, 2022 and complete: (a) either 19CS or 19CSVR, (b) the follow up demographic survey, the (c) follow up chatbot exam, and (d) were employed by Evergreen Industrial Services (EIS), CleanHarbors Industrial Service (CleanHarbors), PSC Industrial Cleaning (PSC), or HydroChem Industrial Services, (HydroChem).
All four of these contracting organization work within the same sector of industrial cleaning. All four contracting organizations work in and around confined spaces, but do not have a sole focus on confined space work as other organizations’ might. Having four contracting organizations as the supply line for learners for the sample will ensure that the sample population isn’t marred by variability of confined space knowledge or expertise that could sway the analysis. All four contracting organizations also subscribe to basic confined space awareness training which will ensure a baseline understanding by the entire sample population. This will ensure at some point in the learners’ past training/work experience, regardless of contracting organization, the learners have taken the exact similar confined space awareness training.
Experimental Design
In 2020, HASC performed a study that was similar to this one except for two factors (Health and Safety Council, 2020). The first difference between this study and the 2020 study is that HASC did not capture learner demographic information for their 2020 study. Secondly, the learner sample population consisted primarily of self-selected learners. Learners were asked if they wanted to take a VR course or the CBT course. According to Keppel “the assurance that each member of the population has an equally likely chance of being selected for the experiment. If these conditions are met, we will be able to generalize the result of our experiment to the population” (p. 17). This 2022 HASC study is focused on creating a stronger experimental design both by capturing learner’s demographic information and analyzing it, but also by ensuring that the sample population is a true random sample from a group of unique organizations in one sector of the energy industry.
Both 19CS and 19CSVR courses include industry standard information provided via articulate storyline files with narration, an unscored pre-test to inspire engagement, animations, interactive scenarios, unscored in-course knowledge check quizzes with remediation for incorrect answers, a scored 20 question final exam, and a chatbot style quiz delivered one week after the course completion. All of these assests have been developed by HASC’s eLearning Development team which has over ten years of experience working these subjects and HASC’s learning community.
The scores in the final exam will not be analyzed as a part of this study. The scores on this chatbot quiz that are delivered seven days after the course completion will be used as a point of quantitative comparison for this study. This will help to focus the analysis on “long-term” memory development in a manner similar to other studies on long-term memory development (Karpicke & Roediger, 2008). The chatbot quiz will be delivered to the learner’s smart phone seven days following the completion of either CBT course. All participants within this study will also provide demographic and contact information via a survey to solicit demographic information.
As stated previously, both 19CS and 19CSVR include the exact same CBT content delivered the exact same way. However, 19CSVR includes two, 3-5 minutes VR exercises designed to allow the learner the opportunity to practice procedures learned in the CBT course in an immersive virtual world using an HASC VR headset. The VR exercises are developed within the CBT so that they occur near the theoretical information that the exercises support. The learner will not have to leave the testing area or leave the CBT workstation to take the VR exercises. The questions in the demographic survey will consist of the following:
• questions to determine the learner’s motivation, expectations, and satisfaction level with the course
• questions to determine the learner’s age, race, gender, years of experience in the industry, years of experience working with confined spaces, academic achievement level, and technical expertise.
The 19CS course is approximately 60 minutes in length. The 19CSVR course that includes VR exercises is an additional ten minutes for a total approximate course length of 70 minutes. The chatbot quiz is seven questions and takes less than five minutes to complete.
Variables
The independent variables for this study are: (a) the confined space awareness CBT course, (b) the VR exercises, and (c) the demographic information of the learners. The dependent variable for this study are the grades from the follow-up chatbot style quiz that the learners took one week following the completion of the course.
Instrumentation and Procedures
Learners, regardless of condition will use similar computer terminals for delivery of the CBT. HASC has over 800 computer terminals for CBT course delivery. All are similar and proctored by HASC proctors at all times during the delivery of the CBT courses. All subjects regardless of condition will be proctored by HASC’s proctors and will be allowed the opportunity to ask questions regarding delivery of the CBT courses.
Learners who take 19CSVR will be proctored and assisted in the use of the VR equipment. During the month-long data collection period, proctors will be trained and monitored by the primary researcher to ensure that the information provided to the learner will be the same regardless of which proctor might interact with the learner. HASC proctors are trained to provide oversight, enforce lab rules, and aid learners in the use of the CBT delivery PC and the use of VR equipment, without giving benefit to the learner that could change how they might answer questions within the CBT, within the VR exercise or after the course is complete when the learner completes the chatbot quiz. Proctors also help ensure that learners do not benefit from other learners’ information or courses.
The HASC CBT lab has individual PC workstations that are isolated from neighboring PC workstations by partitions. Learner integrity is monitored by an established and audited “HASC learner verification process” which verifies learner identification against a picture and other biometric data located in the HASC learning management system (LMS). The HASC LMS was developed by HASC and the HASC researchers will have access to all test and course information through this application. The HASC learner verification process includes three standard identification checks and spot checks throughout the learner’s completion of any CBT. The three times throughout the process of completing the course include: (a) at the registration desk when the learner enters HASC's building, (b) when the learner arrives at the CBT workstation, and (c) when the learner completes the CBT. There are also spot checks by proctors throughout the course delivery process where proctors check that the learner and who checked in and started the course is the same learner who takes the course by comparing the learner at the workstation to the picture of the learner in the stored in the LMS.
The VR exercises that are a part of the 19CSVR course will be administered at the same workstation that the CBTs are taken. Learners in the VR condition will take the VR exercises through an Oculus Quest VR Headset provided by HASC. Demographic surveys will be provided in Articulate storyline after the CBT course is completed regardless of condition. Follow-up chatbot quizzes will be provided via SMS text directly to the learner’s smart phones using an established chatbot technology HASC has used in previous studies. The chatbot quiz will consist of seven multiple choice questions that are designed to judge learner’s long-term memory resilience of the learning objectives regardless of which condition the learner participated in.
Learners will give consent to be a part of this study by accepting a consent form at the beginning of the CBT course delivery regardless of the condition.
Study Measures and Data Collection
The primary measure used for this study will be the chatbot quiz provided one week following the completion of the CBT. The data HASC collects as a part of this study includes both demographic data by way of a learner completed survey delivered via the HASC LMS, and a chatbot quiz delivered via a third-party vendor, MobileCaoch. Learners will also provide consent to use their smart phones for delivery of the chatbot quiz through the use of that same consent form. Learners telephone numbers will be gathered through the use of the demographic survey form. No fields in the demographic survey will be optional. HASC’s researchers have access to all of the data that learners provide via the MobileCoach chatbot test through a secure MobileCoach table that can be accessed via password.
Ethical Considerations
Data collection will not begin until the study is approved by the executive leadership team of HASC. No data collected from this study or from any of the surveys will be disclosed to the public nor to the contracting organizations that the participants work for. Learners names will not be used during the collection, only the demographic data, phone numbers, and chatbot quiz scores will be used during data analysis.
Data Analysis
Upon acquiring data from this study and the study participants, various measures will be used to address the research questions and identify areas that will help provide insight to help answer the research questions. Data from both the survey and the quiz will be imported into SPSS software to analyze and address the research questions. The unique identifier that will be used to analyze the data will be the learner’s phone number. A between subjects factorial design will be employed to help analyze the data. The data will be imported into SPSS and the analysis will include the following:
• descriptive statistics
• Levene’s test of equality of error
• test of between-subjects effects
• estimated marginal means test (without more like video, this may not be necessary)
• Bonferroni comparison (without more like video, this may not be necessary)
The analysis also will also include a summary of demographic and descriptive statistics such as: (a) the mean score on the chatbot quiz, (b) standard deviation, and (c) ranges. To complete this analysis we will use a factorial ANOVA that focuses on comparing the mean score of learners who took 19CS and the mean score of learners who took 19CSVR. Focusing on these two factors will help HASC reject or fail to reject the null hypothesis. Using a factorial ANOVA will allow HASC to determine if demographic characteristics might significantly interact with the learners score on either course.
Limitations of the Research Design
In this study the limitations include the registration process from the contracting organizations. Although HASC has limited the sample providing organizations all work within the industrial services industries, there could be slightly different specialties and different company specific training standards. These different company specific training standards could provide a greater degree of baseline knowledge to the learners in the sample population. Additionally, some of these organizations may have employees who work with and around confined spaces more often than others.
Expected Findings
HASC expects that this proposed study will yield results similar to the 2020 study that compared results from a fire watch CBT to the results from a fire watch CBT with VR exercises which showed almost 10% better recall by the VR condition (Health and Safety Council, 2020; Krokos, et al., 2019). The studies are modeled in a similar manual. The opportunity that this proposed study provides is to determine if the results found in the 2020 study, which relied on self-selection, are found in a study with a stronger experimental design that enforces a random sample population. Additionally, the opportunity that this study provides is the chance to determine if there are any additional interactions between the main effect of long-term memory development with any demographic information provided by the sample population. HASC expects that there will be a significant interaction between "technical experience" and "years of experience" and long-term memory development shown by the learners' grades on the follow-up, chatbot style quiz. HASC expects this interaction effect due to the ubiquity of the material being covered, and the expertise of those learners with greater technical experience and years of experience in the industry.
Ethical Issues
There was no possibility for financial gain for either HASC or the contractors who took part in sending their employees to HASC for training. The learners who took 19CSVR are expected to show better results in long-term memory development after taking the follow-up, chatbot style quiz, but whether taking 19CS or 19CSVR the learner will be fulfilling the compliance and regulatory training demands necessary for safework in the field as mandated by their contracting organization, the refinery site, and federal work safety standards. The researcher and HASC do not have a personal or professional relationship with any potential study participants in the study. The researcher did not have a personal relationship with the authors of the survey instrument used for this study. HASC hopes that the data from this study will reveal practical implications for engaging learners by applying constructivist theory to the instructional design for 3D VR exercises and spaced practice quizzing.
For validity of research, the adherence to ethical protocols is an important consideration. Prior to the data collection phase for the purpose of this study, Institutional Review Board (IRB) approval was obtained from Kansas State University and the research institution. This step will be taken to ensure participant confidentiality and ethical procedures are reviewed for the protection and storage of data collected for use in this study. All participants in this study will receive informed consent forms prior to their survey participation as a part of the course. The informed consent form will state:
1. purpose and description of the study;
2. participant expectations and participation requirements of the study;
3. participant rights, risks, benefits, confidentiality protection; and
4. participant option to leave the study at any time.
Participants were provided details of the precautions implemented to protect their confidentiality during and after the study. Participants were not asked nor provided a means to give information that would identify them except for providing a phone number which will not be linked to their name. Participants electronically signed the consent form prior to participating in the survey collection of data.
Summary
This chapter described the research design and methodology implemented for this proposed HASC VR study. An overview was provided describing an overview of the technology used in this research as well as the setting of the study, participants, data collection tools, data collection process, and data analysis. There was also a section on the limitations of the study as well as information regarding ethical considerations concerning this study and how they were addressed.
References
Karpicke, J. D., & Roediger, H. L., 3rd. (2008). The critical importance of retrieval for learning. Science (New York, N.Y.), 319(5865), 966–968. https://doi.org/10.1126/science.1152408
Health and Safety Council. (2020). Gauging effectiveness of 3D VR for memory retrieval. Retrieved October 10, 2021, from https://hasc.com/uploads/digital/VR/HASC-VR-in-Industry-White-Paper.pdf
Keppel, G. (1991). Design and analysis: A researcher’s handbook (3rd ed.). Pearson.
Krokos, E., Plaisant, C., & Varshney, A. (2019). Virtual memory palaces: immersion aids recall. Virtual Reality, 23(1), 1–15. https://doi.org/10.1007/s10055-018-0346-3
No comments:
Post a Comment