Learner Analytics; The Need for User-Centred Design in Learning Analytics

The interaction and interface design of Learning Analytics systems is often based upon the ability of the developer to extract information from disparate sources and not on the types of data and interpretive needs of the user. Current systems also tend to focus on the educator’s view and very rarely involve the user in the development process. From using HCI methods, we have found that learners want to be able to access an overarching view of their previous, current and future learning activity. We propose that the only way of truly creating a personalised, supportive system of education is to place the learner at the centre, giving them control of their own Learner Analytics.


Introduction
Higher Education gathers an "astonishing array of data about its 'customers'" but has traditionally been inefficient in its data usage [1].The analysis of this data though has th e potential to identify at-risk learners and provide intervention to assist learners in achieving success [2].Therefore, increasingly, student data is being aggregated and present ed to tutors in the form of Learning Analytics (LA) Dashboards.However, the representation and interaction methods being used are often based upon the ability of the developer to extract information from disparate sources and not on the types of data and interpretive needs of the user [3].Making information available and transparent to tutors is only the first step however.Presenting student data back to students, using student centric formats and metaphors could tackle students' inability to access a composite, overarching view of their current learning activity, which can impact on a student's ability to develop creative divergent thinking skills [4].Both conceptual and visual metaphors have historically been used as an effective teaching tool and have been proven to enhance motivation, learning and retention [5] [6].They must however have a high degree of resonance for the learner [5].
A number of projects have investigated the use of LA and information representation/visualisation such as the Open University's Anywhere app which includes a "range of analytics that show how students engage with it" [7], the University of Bedfordshire's student engagement system [8] and London South Bank University's partnership with IBM to "use predictive analytics to gauge if they might be falling behind" [9].However, there have been few studies that have systematically identified sources of pre-existing data and metrics that are currently used within a HE setting and considered the most appropriate way of analysing, representing and making these available to the learner and teacher.More importantly, it also seems that there has been little consideration given to how Learning Analytics can actually be integrated into Learning and Teaching activities.

Background
As part of an on-going project that is identifying factors for consideration in LA, we are conducting a literature review of work relating to LA systems, together with a review of systems in use within academic institutions.The review has currently identified 22 LA related systems, ranging from tools designed to identify 'at risk' students such as Purdue University's Course Signals [10] and the Student Success System [11] which both use traffic light representations, to tools with more specific goals such as engaging and activating students within large lecture theatres [12] or supporting group work by visualising participation [13].The review has so far found that although there are a number of projects investigating the use of LA there appear to be few systems that are available for general use.The review has also highlighted that existing systems are primarily targeted at educators, with only 5 of the 22 systems being designed purely for use by students.Significantly, only 4 of the studies gathered the requirements for the systems directly from students during the development process and only 4 reported an evaluation of the LA tool with students.

Learning Analytics for Teachers
Popular Virtual Learning Environments (VLE) such as Moodle and Blackboard support basic versions of LA e.g.Course Reports and Performance Dashboards.Contextual interviews that we have conducted with educators at Keele University however have shown that these aspects of the VLE are rarely used and do not support the questions that they would like to answer about their students' learning activity.The sessions have also highlighted that there is no easily accessible method for seeing a student's overall level of interaction on all modules, or a way of identifying clusters of students and their usage of specific types of resource; areas that have been identified as having significant potential in identifying students at risk of failing within a module [14].

Learner Analytics for Students
Considering the lack of work focused on the student view of LA, we have used a User Centred Design (UCD) approach with a group of 82 second year Computer Science students to identify GUI metaphors that will engage and motivate them as learners and personalise their own learning experience.Students were tasked with designing LA Dashboards that allow students to review their own progress, display relevant indicators of engagement with their learning and encourage engagement.They were instructed to consider representing data that is already being collected and suggest new sources of data that could easily be collected.As part of the UCD process, deliverables included sets of User Persona, analysed results of requirements elicitation sessions e.g.card sorts, think aloud sessions, and annotated screen mock-ups of potential LA Dashboards (highlighting 5 key features along with objective justifications wherever possible).
A preliminary thematic analysis of the Dashboards from 22 students has suggested that their understanding of LA and their requirements for it are often formed by the limitations of the technologies and systems that they currently use within the University.Of the 117 LA Dashboard features that were proposed, 86% of the students specified features related to HCI/design e.g. the need for accessibility options, device compatibility, suggested layout and display options.This might be due to the background of the students and the content of the module itself but it is interesting to see that the students realise the importance of appropriate interaction and interface design.The latter is something that, from anecdotal evidence, is lacking in the systems currently in place to support their learning.A similarly high percentage of features proposed by the students related to ways of representing a student's progress.These included representations of their attendance, assessment and VLE activity, often in the form of engagement scores, activity meters, progress trackers and comparisons to their colleagues.An associated set of features proposed by 73% of students related to scheduling e.g.coursework hand-in dates and timetabling.Other features relating to resources e.g.module information, links to recently uploaded resources, relevant book availability, were also mentioned, along with suggestions for alternative/simpler communication methods e.g.instant messaging and the ability to contact lecturers directly about feedback from a particular piece of coursework.
Examples of some of the LA Dashboards that students have created can be viewed online (http://bit.ly/LA_Dashboard_egs) along with the features that they represent.Figure 1 below however shows a common representation that was proposed relating to progress and scheduling using a timeline metaphor.The justification for why the student had included the timeline was as follows: My Timeline: Past events which have been completed successfully provide gratification to the user in the form of positive icons such as "thumbs up" or "smiley faces".The past timeline balances with the upcoming events to try and alleviate future workload stress by demonstrat i n g positive success at the same time.
At the moment, some of the information suggested by students to be included on a LA Dashboard is available to them.However, the data is stored in disparate systems across the University or hidden within module content in different, often inaccessible formats.It also requires the students to combine all of the information, interpret it and create their own composite view of their previous, current and future learning activity.Without the development of higher order thinking skills such as metacognition, this would be difficult to achieve.This also requires knowledge and experience e.g.average class marks and the time it takes to complete a new piece of coursework, that they may not have access to.
The design solution suggested by students to this problem can be summarised as follows: (i) Consider and use well known interaction and interface design principles.(ii) Group data together in one place that represents their engagement and progress.(iii) Provide information and functionality that helps them to schedule their learning.(iv) Highlight resources that will support their learning.(v) Offer alternative forms of communication that directly relate to their learning content.

Conclusions
It is clear that HCI is an important factor for consideration when designing LA systems; not just for interaction with the system itself but also when supporting a studen t's access to an overview of their learning.Current systems have tried to tackle some of these issues but HCI and UCD are often being ignored and users i.e. learners are often not included in the development process.This not only causes issues related to usability and accessibility but also means that the features that students want are often missed.It also means that assumptions are made as to how students want to interact, not only with VLE's, but with resources, educators and their own engagement and assessment.So far Learning Analytics has tended to focus on an educator's and administrator's view of a student's learning, mainly in the form of measuring engagement.
The only way however of truly advancing the envisaged "personalized, supportive system of higher education" [15] is to place the learner at the centre, giving them control of their own Learner Analytics.

Figure 1 .
Figure 1.Timeline representation combining progress and scheduling features.

Table 1 .
Features (grouped during thematic analysis) proposed by students (n=22) for an LA Dashboard.