Studying lecturers ’ behaviour patterns in Learning Management System on di ff erent training modalities and knowledge areas

Learning Management Systems (LMS) have become the principal resource for collaboration among lecturers and students in Higher Education. A research line is the analysis of LMS users’ behaviour with the goal of developing new methodological proposals to improve the teaching process. In particular, this paper analyses the lecturers’ behaviour and proposes an alternative vision of teaching evaluation on the use of LMS. Two methods are employed: (1) surveys, to measure the students’ perception on the lecturers’ methodology applied in the LMS; and (2) a statistical analysis of the events generated by lecturers in the LMS. The data have been obtained from degrees that are imparted in different teaching modalities (on-campus, blended and online). These data have been compared over the time period of the two academic years. Among some of the findings of the study, it is observed that students may negatively perceive an excessive monitoring and availability of resources provided by lecturers. Received on 15 June 2018; accepted on 15 October 2018; published on 28 February 2019


Introduction
Thanks to the evolution of e-learning platforms in the last decade, distance learning methodologies in university education such as online and blended learning have been consolidated [1,2].In all of them, the use of online learning tools is essential as they play a fundamental role in the integration of educational platforms at universities.By means of these platforms, usually called Learning Management Systems (LMS), the students can work autonomously, the interaction with other users -whether students or lecturers-is fostered through collaborative tools, new methods for resource management are provided and, above all, new teaching and learning models are consolidated [3,4].
In the last few years, new areas of study are being explored with the common goal of improving the teaching and learning process through the use of LMS platforms [5].An interesting line of research is related Magdalena Cantabella et al. into account in the study, where the educational methodology is evaluated by means of the use of the LMS Sakai 1 , since it is the one adopted in our institution.
The main goal of this work is to discover possible connections between the teaching methodology adopted by lecturers in the LMS and the students' perceptions, which could help to detect possible deficiencies or needs in the teaching process.Hence, the study is structured in the following three stages: 1. Students' assessment of lecturers in all the courses taking part in the study, where four dimensions are evaluated: teaching methodology, available resources, course planning and general overview.
2. Analysis of the data extracted from Sakai to evaluate lecturers' accesses to the LMS, most (and least) used tools and associated events.
3. Finally, the results obtained in the previous stages are compared to detect possible deviations or incidences.
In order to perform these stages, this work resorts to the application of ideas from the e-learning analytic field [6].This emergent field proposes the study of connections between educational techniques, concepts of teaching-learning activities and data analysis, taking into account all the roles involved in the education process.Although the results of this work cannot be generalized since it is focused on a specific institution, it may contribute with some insights into the lecturer's teaching process and remark the differences among the different lecturers' behaviours depending on their training modality or knowledge area.
The rest of the paper is structured as follows.Section 2 reviews some previous works related to the analysis of educational data.In section 3 it is explained the development of our proposal.Section 4 shows the case study for the different faculties in UCAM using real Sakai data and the results from the students' assessment of lecturers.Finally, some conclusions and future lines of work are given in section 5.

Related Work
In the field of Education it is a challenge to define techniques aimed to model knowledge computationally and create services or applications that provide customized recommendations according to student/lecturer profiles.In fact, each student has a different methodology for understanding and managing information, as well as each lecturer uses their own teaching methodology.It 1 https://www.sakaiproject.org/features is a hard task to know the different styles, methodologies and processes related to students and lecturers in order to discover patterns that can help both lecturers to improve the quality of their teaching and students to improve their school performance [7].To help with this task, the amount of data generated by an LMS could be processed and analysed by techniques able to extract behaviour patterns.In the literature there are many studies on the students' use and behaviour within an LMS.For example, in [8] a recommendation system for students is developed based on a combination of association rules, content filtering and collaborative filtering.Another example is found in [2] where asynchronous learning processes are studied through an educational data mining approach using data extracted from the records of an LMS system.More studies focused on the analysis of the students' behaviour can be found in [9][10][11][12], just to mention a few.
However, the analysis of lecturers' behaviours in LMS platforms has not been so well explored.One of our previous work [13] focuses on this area by conducting an analysis and evaluation of the teaching activity in LMS, both from an objective and subjective perspective.The main goal in that work was to compare the own lecturers' perception about their performance in the LMS with the real data gathered from the platform, showing that lecturers tend to overestimate the number of activities they perform in the LMS.In [14] a study is performed to analyse and understand the perceptions of teachers and students about an LMS.The aim of this research is to anticipate possible problems and help in the improvement and construction of a productive learning system.Another work that takes into account the behaviour of lecturers is presented in [15].In it, the authors carry out a study for 40 blended and online modules to detect behaviour patterns and to analyse the design of resources created by lecturers with the aim of improving learning performance.
Finally, it is also noteworthy the work presented in [16], where an algorithm called CCA (Course Classification Algorithm) is proposed to be incorporated into an e-learning platform to analyse and assess its contents.The algorithm defines a series of metrics that are used to provide suggestions and improvements on contents and their quality.These suggestions are obtained by analysing the use and quantity of material available as well as the students' use and assessment of such material.With this algorithm lecturers can improve contents and methodologies based on these suggestions.The research line in this work may be an excellent complement to augment the knowledge to improve the use of LMS, not only by focusing on students' and lecturers' behaviours but also in the quality of the academic resources.

Materials and methods
This study has been performed taking into account the data related to the lecturers belonging to the following faculties at UCAM during the academic years 2015/16 and 2016/17: Polytechnic School, Physical Activity and Sport Sciences, Health Sciences, Law and Business, Social Sciences and Communication and Postgraduate Studies (i.e., Master's degrees).Table 1 shows the numbers of lecturers in the three training modalities involved in the study, namely on-campus, blended and online.Observe that not all the faculties offer the three modalities, as for example in the case of Physical Activity and Sport Sciences where only on-campus degrees are available.The lecturers belong exclusively to each modality.
In order to perform this study, two different types of sources of data have been analysed: (1) LMS events generated by the lecturers and (2) students' evaluations of lecturers belonging to the aforementioned faculties.Firstly, let us explain the LMS tool events to be analysed in the study.In particular, we have used Sakai as the reference platform for the study, as it is the one adopted in our University.In this work we have classified these tools into two groups, content tools and student monitoring tools, as follows: • Content tools: Lesson Builder and Resources (including multimedia resources).
Note that the Lesson Builder is a tool that allows lecturers to use web-based templates to distribute the contents of each unit and create a learning methodology (e.g., first it is shown the text resources explaining the whole lesson, then additional and recommended readings, then some multimedia resources with the most important highlights of that unit, then some related assignments, etc.). Figure 1 shows an example of a typical organization for a learning unit using the Lesson Builder tool.In the figure it can be observed the goals, resources and some videos for a C programming unit.
Secondly, the students' evaluations of lecturers are aimed to evaluate four dimensions in the lecturers' performance in LMS: Methodology, Planning, Resources and General Overview.These dimensions are graded by the students following a Likert-type scale from 1 (strong negative perception) to 5 (strong positive perception).The students are provided with a link to anonymously fill in an electronic questionnaire about these dimensions during the last month of each academic quarter.
Next it is described how the data have been gathered for the study.In the first place, the lecturers' events in the LMS are collected by means of a web application called OnlineData [17].This application has been developed as part of our previous work, and, in a nutshell, it enables centralizing and configuring all the interesting metrics related to LMS tools by directly querying the LMS underlying database.These metrics are then presented in different visual forms and with different level of detail.The metrics not only consist on the number of elements created by the lecturer, such as number of uploaded audiovisual resources, number of assignments, number of sessions of videoconference, etc., but also monitoring metrics such as average response time to students' posts in forum, assignments, private messages, etc.The interested reader could refer to [17] for further information on OnlineData.The total number of LMS tool events generated by lecturers involved in this study sums up to 1,600,574 events.In the second place, the students' evaluations of lecturers was conducted by 1786 students in on-campus modality, 194 students in blended modality and 1097 students in online for the academic year 2015/16 and by 1846 students in on-campus, 244 students in blended and 1024 students in online in the academic year 2016/17, respectively.
Finally, the following data analyses have been performed to evaluate lecturers' behaviours: • Event ranking: This analysis explores the events performed by lecturers to identify not only the most frequent events performed by them, but also the absence of particular events.In this manner it is possible to detect the most (and least) used tools by lecturers and what actions they perform in those tools.
• Time profile of lecturers' log-ins: The intention for this analysis is to evaluate the lecturers' login events in order to detect their monthly and weekly connection behaviour.In this manner it is possible to identify specific periods with high or low lecturers' activity in the LMS.
• Results of the students' evaluations of lecturers vs. Sakai events: Here it is analysed the students' perceptions on the lecturers' performance according to the aforementioned four dimensions with respect to the number of events related to content and monitor LMS tools.
These data analyses have been performed by means of the OnlineData tool whereas the visualization of the results has been performed by means of QlikView [18].

Results
This section explores the results of the analyses proposed in Section 3. Firstly, a ranking on the generated events is studied for the different faculties according to the on-campus, blended and online  modalities.Secondly, the lecturers' time profile trends are analysed on a weekly and monthly basis.Finally, it is presented a comparison of the actual lecturers' performance in the LMS with the students' perception of such performance.

Event Rankings
Given the large amount of information to be shown, the following approach has been adopted to show clear, simple and proportional information according to the number of lecturers teaching in each modality.Thus, for each event E in the LMS a variable named "mean of event E per lecturer" has been created consisting in adding the occurrences of that event generated by lecturers of each modality in both academic years and weighting it by the number of lecturers in each modality in those years.Then, to select the most relevant events, instead of elaborating a top raking of the most frequent ones (which may be biased by the difference among the total number of lecturers in each modality), we have used as a threshold the median value out of the set of means of these variables for each modality, having removed first the irrelevant events (i.e., those with a mean value less than 1 event/lecturer).Finally, an event is considered as relevant if its "event per lecturer" mean is greater than its corresponding median value.Event ranking for the on-campus modality.Figure 2 shows the event ranking for the on-campus modality in the academic years 2015/16 and 2016/17.The most common events for this modality are "Announcement read", "Assignment read", "Lesson Builder: Update lesson", "Lesson Builder: Page read", "Resource download", "Resource create", "Assignment grade" and "Assignment saved as draft".The most relevant findings are commented next: • The number of these events in the faculty of Social Sciences and Communication is reduced from the academic year 2015/16 to the 2016/17, in a complete contrast to postgraduate studies where the number of events increases for the same period.For the rest of the faculties, the number of these events remain generally unchanged during this period.
• In general, the events related to the Assignment tool do not experiment significant changes from one academic year to the next, except for the faculty of Health Sciences where a notable increase is detected for the academic year 2016/17.
• The faculties more "actives" (i.e., the ones holding the greatest number of events per lecturer) are the faculties of Health sciences and Physical Activity and Sports sciences, whereas Postgraduate Studies is the less active faculty.
• It is noteworthy the occurrence of events related to the Lesson Builder tool in this modality, since it is a tool mainly addressed for e-learning modalities.
Event ranking for the online modality.Figure 3 shows the events ranking for the faculties of Health Sciences, Polytechnic School and Law and Business for the online modality and the 2015/16 and 2016/17 academic years.For this modality there are more significant events to highlight than in the on-campus modality, explained next: • There is an increase in events related to the Lesson Builder in the Polytechnic School for the 2016/17 year.However, with respect to the Assignment tool, this faculty held less events in the 2016/17 with respect to the previous one.Even though, it is observed that the Assignment tool is the most used in this faculty.
• The faculty of Law and Business follows a similar behaviour in all events.The most significant changes can be seen in the "Assignment grade" and "Resource release" events, where there is an increasing trend from the 2015/16 to the 2016/17 year.
• The faculty of Health Sciences shows a stable trend in terms of the number of events generated in both years.It is worth mentioning the increase in the number of "Assignment grade" events from the 2015/16 to the 2016/17 year.
Event ranking for the blended modality.Figure 4 shows the event ranking for the blended modality in the academic years 2015/16 and 2016/17.Among the tools most used in this ranking of events are the Lesson Builder and Assignment.The most relevant findings are commented next: • For the faculties of Social Sciences and Communication and Postgraduate Studies there is a general increase in the number of events from 2015/16 to 2016/17.It may be highlighted the following events with a very significant increment for the Social Sciences and communication faculty: "Lesson Builder: create lesson", "Lesson Builder: update lesson" and "Resource download".
• In contrast, in the faculties of Health Sciences and Law and Business there is a decrease in the number of events from 2015/16 to 2016/17.Note that the events of "Lesson builder: create lesson" decrease almost by half in the Health Sciences faculty and a further decrease is observed in the "Resource download" event for this same faculty.
In addition, for the Law and Business faculty, the drop in events in the 2016/17year occurs at the "Assignment grade" event.
As a summary, it is observed the the tools "Resources", "Assignments" and "Lesson Builder" are the most used in any modality, with some significant changes in the volume of use of such tools depending on the faculty and the academic year.However, no clear reasons have been found about these variations and more research work must be done in this point.On the other hand, the use of the Lesson Builder in the on-campus modality certainly shows that this tool is deemed as very useful by lecturers even for on-campus   degrees, as it allows for a better organization of the course.

Time Profile of Lecturers' Log-Ins
In this section are discussed some graphs that present relevant results for the study of the lecturer's log-ins patterns.
Firstly, it has been studied the total number of connections made by lecturers according to their training modality (regardless of the faculty the belong to) for each month of the year when aggregating the number of log-ins in the academic years 2015/16 and 2016/17.The result is shown in Figure 5, where the following insights may be highlighted: • Lecturers in blended and online modalities have a similar behaviour both in the variation of accesses and in the period of the year that such variations happen.Thus, in February, May and October the activity increases, while in April and December it decreases.It is observed that almost  no connections are recorded in August, as it is the holiday month in the University.
• Lecturers in the on-campus modality have a partially different behaviour with respect to the other two modalities.It is worth noting that their activity in the LMS continues to increase from May to June, unlike the rest of the modalities; and from October to November it remains stable, while there is a decrease in accesses in the online and blended modalities.• It is observed a steep increase in the number of lecturer's accesses in the blended modality in October, surpassing the activity with respect to the on-campus modality.
A more deep analysis of this annual trend has been performed by taking into account the faculties in the study.As a result, some further findings are discovered for online and blended modalities.Figures 6(a • For the blended modality, it is shown that the number of connections decreases from 2015/16 to 2016/17, except for the Law and business Faculty, which improves its log-ins considerably. In the 2016/17 year (see Figure 6(d)), it is worth highlighting a greater stability in the lecturer's accesses with respect to the previous year, thus observing an improved follow-up activity between February and June.
Finally, we have searched for log-in patterns attending to the days of the week.While there is no significant trend for lecturers in the online modality, some interesting insights have been discovered in the rest of modalities.Thus, Figures 7(a Thus, an improvement in lecturer tracking is identified in 2016/17 with respect to the previous year.A deviation from this increasing pattern can be observed in the Physical Activity and Sports faculty, which in spite of a general increasing in the number of weekly connections, there is a notable decrease in the log-ins from Wednesday and Friday.
• In the case of the blended modality, it is also detected an increase in the activity of the lecturers in most of the faculties, being the Law and Business faculty where the greatest increase can be observed.On the other hand, the Social Science and Communication faculty does not follow this pattern, since a notable decrease in the number of connections is observed.
• When analysing peaks of connections, in the online modality it is observed that generally Monday and Thursdays are the days with a higher number of connections in both academic years.A Summarizing, it is observed that the number of lecturer's connections follows a similar pattern for elearning modalities, with some difference for the oncampus modality.The most active months in the elearning modalities (February and October) coincide with the starting of each quarter, therefore it suggest that lecturers in e-learning degrees usually upload all the materials and tasks at the very beginning of the course, and then they keep a monitoring profile during the rest of the course.Contrarily, lecturers in on-campus degrees prefer to have a more continuous access to the LMS to upload the materials and tasks in a more escalated manner.Regarding the weekly connection patterns, lecturers usually prefer to access the LMS at the very beginning of the week and then some significant follow-up is done at the end of the weekdays.

Comparison of the students' assessment on lecturers' methodology vs. Sakai Events
This section compares the students' perception on the teaching methodology with the Sakai events generated by lecturers in order to identify possible deficiencies.Firstly, a statistical analysis is performed on the results of the surveys to obtain the mean and standard deviation of the four dimensions being evaluated (Methodology, Planning, Resources and General Overview), as explained in Section 3. Secondly, these surveys results are contrasted with the number of events stored in Sakai as follows: • The "Resources" and "General Overview" items in the survey are compared to the "Resource Events" set, which is composed of Sakai's events related to the "Resources" and "Lesson Builder" tools.
• The "Methodology" and "Planning" items in the survey are compared to the "Tracking Events" set, which is composed of Sakai's events related to the events of the "Forum", "Announcements", "Calendar", "Private messages", "Assignments" and "Tests" tools.
The comparison is shown in Table 2.The values in the table show the differences in the results obtained between the 2015/16 and 2016/17 years grouped by faculty and modality.For each of the dimensions, the green colour indicates an increase of the value in the item with respect to the previous year and the red colour indicates a decrease of such a value.The actual values for each item may be consulted in Table 3 and Table 4 in the Annex section.
The results obtained after comparing both years show that in most faculties the dimensions assessed by the students are improving (except for the Law and Business and Social Sciences and Communication faculties) and the events performed by the lecturers are increasing (except for the Polytechnic School and Social Sciences and Communication faculties).The highest increase in student evaluation of the teaching methodology is seen in Postgraduate Studies for the oncampus modality and the most significantly decreases is found in the Law and Business faculty for online modality.

Conclusion and Future Work
In this paper a study has been presented to analyse possible connections between the methodology applied by lecturers in Learning Management Systems (LMS) and the students' perception of such methodology in order to detect possible deficiencies and improve the teaching process.LMS events generated by lecturers during their activity in the platform are analysed and compared to the results obtained from a lecturer assessment survey conducted by students.The Sakai LMS has been used as the specific learning platform during the academic years 2015/16 and 2016/17 in several faculties at the Catholic University of Murcia (UCAM) which offer three training modalities, namely on-campus (face-to-face), blended and online.
The analysis of the results indicates that the most used tools by lectures in any modality are those dealing with resources and assignments, along with a specific Sakai's tool called Lesson Builder that allows them to organize the course contents in the LMS.However, the use of these tools varies greatly depending on the year and faculty being studied.Regarding the trend of lecturers' log-ins to the LMS, it has been observed that the e-learning modalities (blended and online) follow a similar pattern, in which lecturers prefer to prepare all the materials at the beginning of each course and then they connect to monitor the students' activities.On the other hand, lecturers in on-campus modality are more keen on keeping a continuous labour of uploading materials as the course progresses.Finally, a comparison between the students' assessment of lecturers' methodology with the real performance of such lecturers in the LMS has been performed.The results of this comparison in terms of students satisfaction with teaching quality decrease slightly from 2015/16 to 2016/17.This lessening is due in part to the growing demands of the students each year and the fact that the students are overwhelmed with too many contents and assignments for each course.
Finally, it is worth mentioning that although LMS events generally increase from one academic year to the next, the results for all the different evaluative items drop slightly.However, this slight decrease in all the evaluative aspects is striking, especially when the lecturers' performance in the LMS has augmented with respect to the previous year.In order to determine the reason for this decrease in the the students' assessment results, several interviews were carried out with a group of students of each faculty.In these interviews, the students justified the reasons for their scores and the conclusions obtained from these interviews indicate that they feel overwhelmed with contents and monitoring tasks, suggesting less resources and assignments but of higher quality.
Three future lines follow this work.Firstly, a study is necessary in the detected differences of behaviours with respect to the most used tools depending on the lecturers' knowledge area and academic year.Secondly, an interesting line is to perform a detailed study of the students' perception of the quality of each specific content provided by lecturers as a complementary source of data to detect good teaching practices.Finally, Big Data technologies will be used to detect and identify the lecturers' behaviour patterns in real-time at the

Annex
This Annex contains the detailed values of the survey taken by students measuring the four dimensions for lecturers' assessment, namely Methodology, Planning, Resources and General Overview, following a Likert-type scale from 1 (strong negative perception) to 5 (strong positive perception).
Here it is shown the mean values and the standard deviation (in X S format, where X is the mean value and S is its correspondent standard deviation) for each of these dimensions.It also shows the actual number of events for the "Resource Events" set, which is composed of Sakai's events related to the "Resources" and "Lesson Builder' tools'; and for the "Tracking Events" set, which is composed of Sakai's events related to the events of the "Forum", "Announcements", "Calendar", "Private messages", "Assignments" and "Tests" tools.
Studying lecturer's behaviour patterns in Learning Management System on different training modalities and knowledge areas

Figure 1 .
Figure 1.A screenshot of the Lesson Builder tool in Sakai showing a typical organization for a learning unit including goals, resources and videos.Text and images are intentionally blurred for the sake of privacy.
Figures 2,3 and 4 show for each training modality a comparison of these relevant events (in percentage) grouped for each faculty and paired for the academic years 2015/16 and 2016/17.Let us now see the results in detail.

5
Studying lecturer's behaviour patterns in Learning Management System on different training modalities and knowledge areas EAI Endorsed Transactions on e-Learning Online First

Figure 2 .
Figure 2. Ranking of events per lecturer (in percentage) related to the on-campus modality for the academic years 2015/16 and 2016/17.

Figure 3 .
Figure 3. Ranking of events per lecturer (in percentage) related to the online modality for the academic years 2015/16 and 2016/17.Note that only faculties with online degrees are shown.

Figure 4 .
Figure 4. Ranking of events per lecturer (in percentage) related to the blended modality for the academic years 2015/16 and 2016/17.Note that only faculties with blended degrees are shown.

Figure 5 .
Figure 5. Annual trend in lecturers' log-ins for each training modality.

Figure 6 .
Figure 6.Annual trends in lecturers' log-ins for faculties with online and blended modalities.
) and 7(b) show the annual connection trends for lecturers in online modality in 2015/16 and 2016/17 years, respectively.Likewise, Figures 6(c) and 7(d) show the results for lecturers in blended modality for the same period: • Regarding the online modality, it can be seen a similar behaviour for the three faculties (Polytechnic School, Health Sciences and Law and Business) in both years, where the maximum connection peaks occur in February and October, and including April for Law and Business.In the rest of the months, the number of connections suffers a continuous decrease in the three faculties.In general, the total number of connections increases slightly in 2016/17 with respect to the previous, but for the Polytechnic school in October, where a decrease of connections from 2015/16 to 2016/17 is detected.

8 MagdalenaFigure 7 .
Figure 7. Weekly trends in lecturers' log-ins for faculties with on-campus and blended modalities.
similar trend takes place in the blended modality, except for the 2015/16 year (Figure 7(c)) when Friday replaces Thursday as one the days with more connections.This pattern does not hold for the Postgraduate Studies in the online modality in both years(Figures 7(a) and 7(b) ), where a more stable teaching follow-up is observed.

Table 1 .
Numbers of lecturers distributed for each faculty, training modality and academic year.

Table 2 .
Comparative between the results of the students' assessment of lecturer' methodologies and Sakai events generated by lecturers.

Table 3 .
Detailed values of the lecturer assessment items surveyed (Part I).
14 Magdalena Cantabella et al.EAI Endorsed Transactions on e-Learning Online First

Table 4 .
Detailed values of the lecturer assessment items surveyed (Part II).
15 Studying lecturer's behaviour patterns in Learning Management System on different training modalities and knowledge areas EAI Endorsed Transactions on e-Learning Online First