Empirical evidence of the game-based learning advantages for online students persistence

The paper presents the empirical results obtained from a study conducted on a game-based online course that took place in 2014 with 47 participants. The study evidenced the benefits of the learning games mechanics on learners’ willingness to continue the course. Assuming the interest for the subject of the course as a fundamental condition for student persistence within the course, it is shown how it can be significantly enhanced by the presence of both ludic and narrative game-based elements.


Introduction
Dropout in e-learning courses is a well-known problem, consistently studied in the literature.The consequences of dropout for the online courses are critical not only for those who abandon the course, but also for those who remain and for the whole organization.Some aspects of the dropout phenomenon have been clearly understood, like, for instance, its appearance very early in the process of learning, in the very first weeks of the course [1].Analysing the causes of dropout, however, is indeed a complex task, and there is not a general consensus or a unique approach in the literature.Reviews by Lee and Choi [2] and Hart [3] summarize years of research on that field.Lee and Choi individuated factors that, according to the scientific community, influence students' decision to dropout, and classified them into three main categories: (a) Student factors, (b) Course/Program factors, and (c) Environmental factors [2].Each of these categories contains many different approaches to the dropout problem, and Storrings argued that there is no significant pattern in identifying variables that cause or contribute to distance education attrition within the empirical literature [4].
Although this attention, the problem is still very actual, especially with the advent of Massive Online Open Courses (MOOCs), whose great popularity is often accompanied by less accuracy for the pedagogical aspects, resulting in even higher rates of abandon of the courses [5].
Lee and Choi classified the strategies to overcome the dropout factors in the following three categories: (a) understanding each student's challenges and potential, (b) providing quality course activities and well-structured supports, and (c) handling environmental issues and emotional challenges [2].
Some authors suggested the improvement of gamebased learning to prevent dropout (for instance [6]).Their main argument was that game-based learning addresses some of the possible causes of dropout, like for instance the lack of motivation, and the sense of frustration [7].However there is a scarce empirical research finalized to study the impact of a game-based approach on students persistence.

The Game-Based Learning Course
In May 2014 a 3 week online game-based course, titled "Strategic Soft Skills", with 47 students, was organized by Link Campus University (Rome, Italy), using the Open Educational Resources of an European project, titled "Professionals in Touch: New non-routine skills via mobile game-based learning (INTOUCH)" .
The topic of the former INTOUCH project was the development through mobile game-based learning of nonroutine skills for Small and Medium-sized Enterprises (SMEs) [14].Ten non-routine skills were addressed: Communication; Planning; Conflict management; Openness to change; Decision making; Teamwork; Flexibility; Strategic thinking; Initiative; Learning and improvement.
At the end of the INTOUCH project a kit of 30 serious games (three for each of the aforementioned non-routine skills) was made available, freely accessible through the Internet.Thanks to these Open Educational Resources it was possible to build a structured on line course.The selfpaced course "Strategic Soft Skills" was formed by a subset of 20 serious games (two for each non-routine skill), chosen as the best ones according to the INTOUCH evaluation [14].Each serious game, previously developed for mobile devices, was embedded in a Shareable Content Object Reference Model (SCORM) learning object to build the index of the new course and track the completion to monitor student progress.Students had three weeks at their disposal to complete the "Strategic Soft Skills" course, and the certification of attendance was obtained if all 20 games were completed with a correctness score above 70%.
The 20 game-based learning objects were designed according to the same scheme, made of an opening scenario, a problem-based situation presenting the aim of the learning object, a central block of interactive frames where users were asked to choose among different options, and the last part of the game showing the closing scenario, the score, and giving feedback to the students.
The central frames were developed according to the following types of interaction:  Branching story: the story develops in different ways according to the choices made by the user and the final feedback and evaluation are the result of the combination of the choices. Interactive map: the user can choose three characters to talk to.Basing on the obtained clues, the player can choose one of the three available alternatives.
Evaluation is based on the final decision and on the choice of the characters. Multiple choice: the user has to help the main character with three different decisions in a limited time frame.In the first decision point only three out of the five listed options are correct, in the second one only two, and in the third one only one.The final score and the feedback depend on how many correct answers the user chooses. Quiz: the player has to try to correctly answer three related questions, getting immediate feedback on the answer to each question and a summary at the end of game.Evaluation is based on a combination of the number of correct answers with the time taken to answer. Task simulation: the player has to prioritize three tasks in order to achieve a goal.Each task is associated with a question to be answered.The score is determined from the number of correct answers and from the order the user chose to prioritize the tasks.
The game-based learning objects were all referred to an archetypal working context well known to the SME's employees, namely a small company composed by eight characters that were described in terms of their personal information, company role, a short bio, and some other charming and funny details such as star sign, hobbies, events from their past, and photographic illustrations showing something weird about them.
The kit of game-based learning objects was developed taking in account both the ludic and the narrative components, that are commonly considered the main features of game-based learning products [15] [16].
Even though challenges are not that complex, learning objects design tried to respect requirements to be relevant, explorative, emotive and engaging for the learners.This was achieved by inserting challenges of mastery and comprehension, together with strategy, so that learning objects become real living puzzles, engaging and challenging, with a perceived risk of failure to prevent boredom.Serious games were thus highly ludic thanks to their gameplay and interactions.
The narrative within the learning objects is developed giving a short background story in the opening scenario, then it is influenced by users' action in the central frames, and ends up with the closing scenario.An effort was made to connect contents to learners' work experiences, in situations and contexts characteristic of day-to-day activities.The serious games, although short and simple, have thus a solid narrative structure in order to engage players, make them recognize narrative patterns referred to their work activities, and give them the right balance between fantasy and real working context situations.To evaluate the impact on students persistence, the set of 20 game-based learning objects was divided in two parts and an intermediate questionnaire was administered between the first and the second part of the course.A first set of 10 learning objects (1 for each non-routine skill) was delivered in the first half of the course.Learners were free to choose the order of fruition, and only after the completion of all the 10 proposed learning objects they were given the possibility to access the second part of the course, formed by another set of 10 learning objects.A mandatory structured questionnaire was proposed to participants after the completion of the first half of the course.

Procedure
The interest for the subject of the course is considered a primary element influencing motivation and preventing from dropping-out.It is then interesting to notice how the development of the course in terms of its game mechanics can influence the causal relationship between the interest for the subject and the willingness to continue the course.
The complete causal model summarizing the relationships among the measured variables is indicated as Model A and is illustrated in Figure 1.In the complete model the relationship between the interest for the subject of the course and the willingness to continue the course is partially mediated both by the satisfaction for the ludic component of the course, and by the satisfaction for the narrative component of the course.The complete Model A will then be confronted with two alternative models, named Model B and Model C, where respectively the mediation of the ludic component (path 1) and the mediation of the narrative component (path 2) will be suppressed to establish which causal model better explains the data.

Measures and results
This section contains an illustration of the methodology that was adopted in the present study and the results that were obtained: a description of the sample; the measures and the statistical analyses that were adopted, with the final results.

Participants
The target sample attending the course consisted of 47 people of 3 different SMEs operating in different business sectors (ICT, business support, education/training).The SMEs were selected on the basis of their willingness to participate in the study.Link Campus University subscribed a contract with the three SMEs, and the management selected people to take the free online course.Work positions were: 6 managers and 41 employees.In total 26 were males (55%) and 21 were females (45%).The mean age was 39.82 years (SD = 9.50).

Persistence in the course
The dropout rate was very low for the course (6.4%) if compared with common dropout rates for e-learning courses, especially for self-paced solutions, with no peercollaboration or tutorship, like the considered one.47 students were signed at the beginning of the course, 2 students (4.3%) abandoned during the first half of the course, 1 more student abandoned during the second half (2.1%).The interim questionnaire was completed by 45 students and its results are reported below.As a preliminary analysis, skewness and kurtosis were checked.Overall, all variables showed to conform to the normal distribution.The correlation matrix of all the variables measured by the questionnaire was then calculated.Table 1 reports correlation coefficients of (a) the willingness to continue the course, (b) the ludic component, (c) the interest for the subject of the course, and (d) the narrative component.All path models involving the aforementioned four variables were analyzed with LISREL, using maximum likelihood estimation procedures [17].For each tested model χ 2 is reported, as an absolute fit index (good fit between zero value and two times the degrees of freedom).Three more fit indexes were also reported: the non-normed fit index (NNFI); the comparative fit index (CFI); and the root mean square error of approximation (RMSEA).Higher CFI and NNFI values (in the range from 0.97 to 1.00 for a good fit) and lower RMSEA values (in the range from 0.00 to 0.05 for a good fit) are assumed to evaluate model fit [18].Finally, the R-square percentage of variance of the willingness to continue the course explained by each model was reported, to estimate the completeness of the considered set of predictors.Table 2 reports the results of the path analysis for the three tested models with the levels of significance of the causal paths (p-values) indicated in the table footnote.The comparison of the alternative nested models was conducted analyzing the differences of the χ 2 values between the complete Model A, with less degrees of freedom, and the more parsimonious ones (i.e., Models B and C), with one more degree of freedom each, due to the suppression of one causal path (path 1 or path 2).The complete Model A is expected to have a better fit, that is a lower value of χ 2 , while Models B and C are expected to have higher values of χ 2 because, as a rule of thumb, the use of more degrees of freedom corresponds to a worse fit.However it must be established if the loose in fit of Models B and C is not so significant as the advantage of using a more parsimonious model.This comparison is made through the distribution of the difference of the χ 2 values between the more parsimonious Models B and C and the complete Model A. Choosing for the significance a cut-off of p = 0.01, the corresponding χ 2 distribution for one degree of freedom (the one gained passing from Model A to Models B or C) has a critical value of χ 2 = 6.64.If the difference of the χ 2 values between two nested models is significant (χ 2 > 6.64), this implies that the complete Model A explains the data better and must be kept; on the contrary, if there is no significant difference between two nested models (χ 2 ≤ 6.64), this implies that the more parsimonious model explains the data equally well compared to the complete model and must be preferred for its simplicity.Table 3 reports the results of the comparison of the fit of the three tested models with the level of significance of the difference between complete and nested models indicated in the table footnote.Both Model B and Model C explain the data equally well compared to the complete Model A (p > 0.01) and must be preferred being more parsimonious.Between the two, Model C exhibits a better data fit (χ 2 and RMSEA) and is also more explicative in terms of percentage of variance of the willingness to continue the course.For these reasons Model C is the best one to be preferred among all three tested models.

Causal Models Analysis
For the selected Model C the effect of the interest for the subject of the course on the willingness to continue the course is made of two contributions: a direct effect (path 3) and an indirect effect through the ludic component (product of path 1 and path 4).The numerical values are reported in Table 4 with the levels of significance in the table footnote.

Conclusion
A significant positive causal relationship was proved among game mechanics and persistence in attending the course.The present study can thus be considered an empirical evidence of the game-based learning advantages for online students persistence.For all the tested models, the R-square values of the willingness to continue the course resulted to be very high (about 90% of the variance explained).This can be seen as an overall confirmation of the right choice of the models' variables and their causal arrangement: the interest for the subject of the course, together with the ludic and narrative components of the serious games, are very significant predictors of the willingness to continue the course.
As hypothesized both the ludic and the narrative components of the considered causal models resulted to significantly influence the willingness to continue the course.Causal paths 4 and 5, linking respectively the ludic and the narrative components to the willingness to continue the course, showed to be significant across all tested models.In particular the ludic influence resulted to be more robust, with values of path 4 above 0.80, while the narrative influence, even though significant, was less pronounced with lower values of path 5 coefficients.
With respect to the complete Model A, where both partial mediations of the ludic and of the narrative components are considered, it was found that more parsimonious Models B and C, where there is only one partial mediation (by the ludic or by the narrative component), must be preferred.In particular, since the causal relationship between the interest for the subject of the course and the ludic component was stronger than the one between the interest for the subject of the course and the narrative component, Model C resulted to be the best fitting one.This conclusion, however, is strictly linked to the considered game-based learning objects and their specific contents in terms of ludic and narrative components and cannot be generalized to every gamebased solution.
The correlation coefficient of the ludic and narrative components, whose value, as reported in Table 1, is -0.14, resulted to be non-significant, indicating their substantial independence.Ludic and narrative components must therefore be considered as separately, and differently, contributing to determine the willingness to continue the course.It can thus be hypothesized that game-based solutions with a predominant ludic component exhibit a behaviour like the one found in our case, with Model C best fitting.In the opposite situation, for those gamebased solutions that are more focused on the narrative component, Model B could be preferred; while those game-based solutions where both the ludic and the narrative components strongly influence the willingness to continue the course are the most suitable to be described by Model A. All these suggestions, however, must be verified and constitutes an indication for future work.
A limit of the present study is represented by the little dimensions of the tested course, both for the low number of participants and for its duration.As a consequence of the small sample, together with the low number of dropouts, the analyzed data are almost exclusively limited to non-abandoning students.In fact, only one student that completed the questionnaire, dropped out in the second half of the course.A rigorous test should be made with courses formed by many participants, since dropouts dramatically increase for massive courses.In this case also dropping out students would be hopefully profiled by the measurements.Furthermore, the recruitment of students for the tested course was obtained through direct contacts with the SMEs whose employees and managers attended the course.On the contrary, huge classes of students without prior personal relationships and not sharing a common workplace, can negatively influence motivation and persistence in the course in a measure that is comparable to the benefits of the game-based approach, thus nullifying the results that were illustrated in this paper.At the same time, the duration of the course should be longer to verify the effective persistence of students descending from the game-based approach.Even if dropout pattern can emerge at the very beginning of a course, having considered a 3-weeks duration is too limited to be generalized to long lasting courses.
One more limit is the extreme simplification of the measurement of the ludic and narrative components of the serious games.comprehensive insight of their role within a serious game should be analyzed with a larger number of indicators.Instead of a generic self-developed questionnaire, with one item for each variable, a validated instrument should be adopted, mapping multiple items to variables through factorization.This simplification is due to the very basic structure of the tested games.Such a simplicity is functional to their original planning for mobile devices.The serious games were conceived to be played anywhere and at any time, at work or at home or even on the way to/from work/home.Games' interface was designed in such a way that a simple touch, or click, was enough to interact, thus enabling one-hand playing.
These limits must be taken into account in view of an implementation of the proposed research concept with other serious games that can present a different balance between narrative and ludic components, and can have a more complex and articulated structure.A perspective for future work is to consider a set of indicators both for the ludic and for the narrative component of the games, like for instance participants' opinion about the rules of the games, the gameplay, the system of badges and rewards, the narrative of the starting scenario, the development of the narration during the games, etc.With such a detailed analysis, covering many aspects of the games, the ludic and the narrative components can be treated as latent factors, studying simultaneously the contributions of the single indicators to each factor (measurement model) and the role of the ludic and narrative components within a causal scheme explaining students persistence (structural model).
An interim evaluation of the game-based course was conducted at the half of the course "Strategic Soft Skills" (during the second week), measuring a set of four variables: (a) the learners' willingness to continue the course, (b) the interest for the subject of the course, (c) the satisfaction for the ludic component of the course, and (d) the satisfaction for the narrative component of the course.

Figure 1 .
Figure 1. the complete causal Model A Both the satisfaction for the ludic component of the course and the satisfaction for the narrative component of the course are hypothesized to positively influence the willingness to continue the course (paths 4and 5).These hypotheses are based on the consideration that both the ludic and the narrative components are significant elements in determining the motivation of the learners.It is also hypothesized that the interest for the subject of the course positively and directly influences the willingness to continue the course (path 3), since the interest can be considered as a natural predictor of the motivation.The complete Model A will then be confronted with two alternative models, named Model B and Model C, where respectively the mediation of the ludic component (path 1) and the mediation of the narrative component (path 2) will be suppressed to establish which causal model better explains the data.

A
questionnaire was proposed to participants, after the completion of the first part of the course, asking them to express on a 10 point Likert scale their degree of satisfaction about (a) the willingness to continue the course, (b) the interest for the subject of the course, (c) the satisfaction for the ludic component of the course, and (d) the satisfaction for the narrative component of the course.
Ludology and narratology are complex and multidimensional concepts, and a more EAI Endorsed Transactions on Serious Games 08 2014 -07 2015 | Volume 1 | Issue 4 | e5

Table 2 .
Path analysis coefficients

Table 3 .
Comparison of the nested models

Table 4 .
Effects on the Willingness to continue the course (Model C) * p < 0.01 ** p < 0.05