Evaluating Factors for Student Success in a Flipped Classroom Approach

This paper describes the application of flipped and blended learning techniques to a Final Year Computer Science course in a UK University. All student interactions with the course are recorded in the institution’s Virtual Learning Environment and a number of metrics of engagement are identified and assessed as potential indicators of success. A statistical analysis of these metrics revealed that although attendance remains a significant indicator of student success in this scenario, consistency of engagement is a more accurate guide. Results show that students with a longest average gap between engagements of 12 days or fewer are likely to achieve the highest grade, while measuring the number of days on which material was accessed or measuring the total time spent engaging with the material are less reliable measures.


Introduction
With the emergence of e-learning technologies and the rapid growth in commercial Massive Open Online Courses [1], traditional learning institutions such as universities have been active in attempting to make use of digital methods to improve the student experience [2]. A common approach has been to adopt what has become known as a blended learning approach [3], most usually interpreted as a combination of traditional face-to-face classes and online delivery, but there is wide variance in how this mix is best achieved. Although most studies into the effectiveness of blended learning report a positive impact on student satisfaction and performance [4], there is contradictory evidence that students prefer traditional delivery [5] and that the "correct" blend differs for individual students [6].
In analysing the key factors of student behaviour that can be used to reliably predict performance, attendance at classes is constantly among the most significant. This is found to be the case both when attendance is mandatory [7,8] and when attendance is optional [9]. Many studies have investigated the reasons for student non-attendance and although some are study-related issues such as prior performance [10], others are environmental and societal [11]. The modern curriculum designer is faced with conflicting challenges -how to encourage students to attend class while simultaneously supporting those who are unable to attend regularly.
The flipped classroom is an active, student-centred approach that aims to maximise the quality of time spent in classroom-based learning [12]. In its purest form, students are provided in advance with the material that would previously have been presented in class, while face-to-face time is freed for problem-solving and other interactive and collaborative activities. The flipped classroom has been shown in studies to have a positive impact on both student engagement [13] and performance [14].
The aim of this study is to investigate the relationship between student attendance and final result in a flipped course where all material is made available to the students regardless of physical attendance. In addition, all student engagement with the material is monitored by the Virtual Learning Environment so that it can be established whether any other patterns of student engagement can be seen to correlate with a successful outcome. In particular, the following research questions will be addressed: 1. Does attendance still provide a good indication of student performance when all of the material is made available outside of timetabled classes? 2. Are there other factors of student engagement that provide a reasonable indication of likely performance in a flipped classroom environment?

Method
The course is a Final Year module in Full Stack Development and is a core module for all students across a range of computing degree programmes. It is taught in blended form across 13 weeks, with all material delivered online through the Blackboard Learn Virtual Learning Environment together with supporting twice-weekly 2-hour practical classes where students receive mentoring and support as they implement solutions to the tasks introduced in the online material. Following the practical classes, additional feedback material is released online. As the group size is large (typically over 200 students), each faceto-face practical class is delivered three times (on the same day), with students each assigned to one of the deliveries. The course is assessed 100% by coursework, with each student delivering an individual full stack application, based on a dataset that they have either sourced online or created themselves. An interim submission in Week 6 allows the students to receive feedback on their choice of dataset and implementation plan for the project, but this carries no marks and is for formative feedback only. All marks are accrued from the final submission in Week 13.
Practical classes take place each Monday and Thursday during the teaching period. The material for each class is made available on Blackboard Learn at the end of the previous class, hence for the Monday class, the material is released on the previous Thursday. The material includes a workbook that presents a step-by-step guide to the subject of the class and a set of related problems and challenges. Some challenges are marked as "Do it now!" and should be attempted by the student at that point in the workbook, while others marked "Try it now!" are more advanced in nature but do not prevent the student from progressing with the material. A set of screencast videos are also made available as part of the class material. These provide a worked demonstration of the material in the workbook, with lecturetype presentations, "live" coding demonstrations and solutions to the "Do it now!" exercises. The expectation is that the students will have studied the workbook and watched the videos in advance of the practical class and that they are able to use the class to obtain assistance on elements that they may have found difficult or for advice on tackling the "Try it now!" exercises. Immediately following the class, a "Feedback Summary" document is released on Blackboard Learn. This is a collection of the key questions and answers raised by students during the session and ensures that any useful tips or key sources of help are disseminated to the entire group -even though they have been spread over a number of physical rooms and over three time slots. Finally, a week after each class, a number of additional videos are uploaded to Blackboard Learn providing worked demonstrations and solutions to the "Try it now!" exercises from the workbook. The schedule of release of material for a pair of Monday and Thursday classes in a given week (N) is illustrated by Table 1. The module had a total enrolment of 205 students, of whom 19 failed to complete the course for a variety of reasons. Those who failed to complete have been excluded from this study which concentrates on the 186 students who made a final submission. The performance profile of these students is shown in Table 2. The overall performance of the group was good, with an average mark of 62% and a standard deviation of 15.9. The highest mark awarded was 95%, while the lowest score achieved was 12%. A total of 118 students (64% of the cohort) scored 60% or above. All submissions and marking were moderated in accordance with University policy.
The aim of this study is to identify, and measure the impact of, factors that contribute to student success on the EAI Endorsed Transactions on e-Learning 07 2020 -02 2021 | Volume 7 | Issue 20 | e3 Evaluating Factors for Student Success in a Flipped Classroom Approach 3 module.
It has long been established that physical attendance at class is a positive indicator in a traditional classroom delivery, but does this still hold in a flipped and blended model where all material is made available regardless of attendance? In addition, are there aspects of a student's engagement with the online material that have a greater impact on their potential for success?
In order to investigate this, Blackboard Learn was configured to generate the set of metrics regarding student patterns of online engagement that are presented in Table 3. These metrics were selected to encompass the time spent engaging with the material, the number of individual engagements, and the consistency of effort over the teaching period. A Pearson Correlation was used to measure the relationship between each student's final mark, their level of attendance at class and their four metrics of online engagement. Attendance was recorded at each practical class by having students sign to verify their presence, thus enabling a differentiation between those who were physically present in class and those who preferred to follow the material online at a time and location of their choosing. The correlation results are presented in Table 4.

Results
The correlation between the final mark achieved and the level of attendance is not as strong as has been observed in other studies in the literature, but this is to be expected as the presentation scheme for the online material does not disadvantage those who choose not to attend. In addition, it can be seen from the table that the correlation between the final mark and the number of days on which a student engaged is much stronger than that between the mark and the number of hours spent engaging with the materialsuggesting that consistency of effort is a better indicator than duration.
To gain an appreciation of the students' typical pattern of engagement with the module, it is useful to examine the average weekly physical attendance in comparison with the average weekly online engagement, shown in Figure 1.
Here, it can be seen that following the initial introduction to the module and method of delivery in Week 1, both attendance and engagement remained relatively static between Weeks 2-7, with average attendance constantly in the range 65-80% (though the general trend was towards a slow decline in attendance), while the total number of online accesses averaged around 1000 hits per week.
Week 8 was designated as a consolidation and catch-up week with no formal classes, hence there is no recorded attendance figure; but online engagement in this week saw a rise to almost 1800 hits representing the highest figure of the semester to date and an increase of 70% over the average between Weeks 2 and 7.

Metric Description
Accesses the total number of times that items of online course material were accessed over the course of the teaching period.
Hours the cumulative time spent accessing online material.
Days the number of distinct days on which a student accessed material.
Gap the longest continuous sequence of days without accessing material.

Figure 1. Attendance vs Online Engagement
When scheduled classes resumed in Week 9, there was a marked decline in both the physical attendance and the level of online engagement. Attendance dropped by almost 50% compared to the average for Weeks 2-7, while online engagement dropped by 32% over the same period. This may be partially explained by coursework submission deadlines for other modules in this period and may also account for the significant rise in online accesses in Weeks 12 and 13, as students caught up with the work in the period after the completion of face-to-face classes and leading up to the assignment submission in Week 13.

Attendance
Attendance at class has long been shown to be a major factor contributing to student success, but the blended delivery model adopted here meant that it was possible for students to gain access to all materials, all feedback and all supporting resources even if they never physically attended class. It is interesting therefore to measure the attendance of this group and to attempt to gauge the impact that attendance has on student performance in this circumstance. Figure 2 plots students' final mark (on the vertical axis) against the percentage of classes attended (on the horizontal axis). Although the plots are reasonably scattered, with many instances of students with low attendance scoring highly and those with high attendance scoring lowly, there is still a moderately positive correlation between attendance and performance. Figure 3 extends this analysis to illustrate the average attendance of students who achieved each of the grade classifications, and demonstrates that those who scored a mark in the 1st Class Honours band (70%+) had, on average, higher attendance than those achieving a 2nd Class Upper result (60-69%); who in turn had higher attendance than those in the 2nd Class Lower band (50-59%).

Number of Accesses of Online Material
The overall attendance rate for each student has been shown to be a moderately positive indicator of success, but a deeper analysis would account for all engagement with the module -whether in a timetabled class or at some other time. As all module activity is conducted through Blackboard Learn, an analysis of the number of occasions on which each student accessed an element of course material catches both those who attend class and those who choose to follow the material online from a location of their choosing. Additionally, this measure will include student activity between scheduled classes, and intuitively should give a feel for the level of effort across the teaching period.
However, Figure 4 shows that the relationship between the final mark and the raw number of accesses is actually weaker than that between final mark and attendance, and the analysis of the average number of accesses by students that achieved each classification grade, presented in Figure 5, shows that students with a minimal passing grade (40-49%) on average accessed the material more often than those who scored the 2nd highest grade (60-69%).

Number of Days on which Material was accessed
An alternative metric is the number of distinct days on which the material was accessed. The previous measure would count (e.g.) 100 accesses in a single day as being equal to (e.g.) 4 accesses per day over a 25-day period, while it seems obvious that the number of distinct days should be a more reliable indication of likely success as it represents a longer period of consistent effort. Figure 6 presents a graph that plots the total number of days on which a student engaged with the module against their final mark. It can be seen from this that the correlation between days of engagement and result is stronger than that between attendance and result and that there are fewer outliers. Figure 7 presents the average number of days of engagement for each of the grade classifications, and again it can clearly be seen that the students who achieved the highest grades tended to be those who had engaged most often with the material -recording 29% more days of access than the average student in the cohort.

Duration of Engagement
The number of days of engagement is a useful metric that exhibits a moderately positive correlation with the final result, but it does not consider the time spent in each session -a short visit of a few seconds counts the same as a longer study session of a number of hours. Figure 8 presents a plot of the total number of hours spent engaging with the material against the final mark achieved, but it is perhaps surprising to find that this measure appears to be a less reliable guide. The correlation between total time spent and average mark is only weakly positive and the chart in Figure 9 reveals that students who scored 3rd Class Honours (50-59%) actually spent longer on average (56 hours) engaging with the material than those who scored a result in the First Class (70%+) range (54 hours). However, it is clear that those students that failed the assessment spent significantly less time engaged with the material than those who passed.

Consistency of Access
The final factor considered was the level of consistency demonstrated by students in terms of their engagement with the material. This was measured by calculating the greatest number of consecutive days during which a student failed to engage. A common pattern of behaviour for weaker students is to be active and engaged at the beginning of a course, only for effort and engagement to diminish as time passes, followed by a sprint of activity at the end as an assignment deadline nears. Although the measure is quite crude (e.g. a single moment's engagement within a long run of inactivity will end the run and result in it being represented as two shorter runs), it is anticipated that it will positively identify those students for whom activity is more "bursty" in nature. Figure 10 presents the longest period of inactivity for each student, plotted against their final result. It reveals that there is indeed a moderate negative correlation between the length of the longest run of inactivity and the final result -i.e. the shorter the longest run, the more likely that the result is good. Figure 11 charts the average length of the longest run for each classification grade, and it can be clearly seen that students that engage consistently over the teaching period are more likely to achieve a better outcome.

Discussion
There are a large number of factors that indicate a student's likely performance on a course delivered by blended learning in a flipped classroom environment. This study has attempted to identify and measure these to produce a model that can be used as a benchmark against which to monitor student engagement and identify students at risk as early as possible so that so that timely interventions can be made. This has long been the policy in courses delivered in a traditional classroom setting, where a continuous period of non-attendance prompts a request for a meeting and some corrective action, but similar schemes are not in common use when all material is delivered online.
When scheduled face-to-face sessions are held, attendance monitoring remains a valuable tool and the results presented here show that even with a blended delivery that does not require students to be physically present in order to receive the material, there remains a positive correlation between attendance and achievement. However, attendance is not the only, or the strongest, indicator of success. This study finds that the strongest indicator is consistency of effort -students who engage with the material on more occasions and with shorter gaps between engagements can be seen to achieve to a higher level than those for whom effort is more "bursty" in nature. This can be seen to have a greater positive effect than the overall time spent in engagement, and indeed the total engagement time is seen to have almost no relation to a student's overall performance. However, it should be recognised that the measures assessed in this study are not independent of each other. Student attendance at a scheduled session will by definition involve engagement with the material on that day and for the duration of the class. In addition, it is perfectly possible for students to engage with the Virtual Learning Environment only for the time it takes to download the material and work thereafter from a local copy, thus circumnavigating the activity tracking facility, but observation of student behaviour in labs suggests that very few (if any) adopt this approach.
There is no doubt that a study of students' work patterns can be used as an indication of their likely success on a course. Delivery using a Virtual Learning Environment that supports activity tracking can produce a wide range of statistics that enable detailed analysis to be carried out. However, the data may contain additional information that the linear regression analysis described here does not reveal. Future work will include the application of machine learning techniques. This will enable a deeper investigation into the relationship between students' work patterns and their overall performance. The ultimate aim is to construct a tool that enables real-time monitoring of student progress and automatic intervention for those for whom engagement is giving cause for concern.