An investigation into the efficacy of avatar-based systems for student advice

Student support is an important function in all universities. Most students expect access to support 24/7, but support staff cannot be available at all times of day. This paper addresses this problem, describing the development of an avatar-based system to guide students through the materials provided by a university student employability service. Firstly, students and staff were surveyed to establish the demand for such a system. The system was then constructed. Finally, the system was evaluated by students and staff, which led to a clearer understanding of the optimal role for avatar-based systems and consequent improvements to the system’s functionality.


Introduction
Effective student support can be vital to progression rates and overall achievement of students [1], [2]. In the university environment, for a typical student, the support role does not rest with a single individual. A student may have a personal tutor, who is the first port of call for issues concerned with their academic progress, and who might refer the student to others to deal with non-academic issues. These others might include counsellors, placement officers, examination officers, accommodation officers etc., each of whom can assume the support role in their specific context. The support role requires good listening skills, the ability to ask the right questions to guide the student, and a comprehensive knowledge of the relevant domain. We investigated the desirability of automating this support role, with a view to providing higher quality interaction with university services at times when face-to-face support was not available, through the use of a Virtual Assistant (VA).
Student support may be categorized into subjectspecific and other roles [1]. Academic tutors are often also the first port of call for non-academic issues. This role, though vital, is rather broad to be used for a pilot study. We therefore decided to focus on developing a prototype system to assist with one highly specific support role; that of the university employability service, which advises students on job applications, CV writing, interview technique etc. If successful, the intention was to roll out the system to other academic and non-academic support roles.
Typically, university services such as counselling, student placements and employability are accessed in the first instance via web pages. There is usually a great volume of static information, together with a few dynamic pages for making appointments or similar. Students often have difficulty in finding the answer to a specific question when trying to navigate these pages [3], [4]. The ideal situation is a face-to-face dialogue with an expert, who can guide the student towards the desired outcome. However, in today's world, students need to access university services at any time of day, and person to person contact is not always possible. In these circumstances, it would clearly be desirable for there to be some kind of artificial agent that could conduct a similar dialogue to guide the student to the required information. A number of rule-based textdialogue systems have been developed and it is possible to combine these with text-to-speech avatar-based interfaces to provide a facsimile of the ideal human-to-human dialogue. The benefits of using a VA include constant availability and the use of natural language communication

EA EAI Endorsed Transactions on e-Learning
Research Article to provide answers to queries or guide the student towards relevant information.

Related work
Studies have been carried out by a number of different research groups on the use of intelligent virtual systems, some of which cover use in higher education for student support. Augusto, McNair, McCullagh and Roberts [5] explored the idea of providing virtual mentoring for students at the University of Ulster. Their focus was on the development and testing of the technology to identify potential uses to support students. Whilst the outcome was limited, the potential for further development was acknowledged. Intelligent agent based systems have also been piloted to support students in the learning of programming languages [6], [7].
The University of Granada [3] developed a virtual assistant for their university website, to provide help for users in finding "not only the information that they are looking for, but also some related information which could be of the highest interest". Their argument was that users often waste time looking through websites when using traditional menu or keyboard methods. According to [3], the need for some sort of intelligent agent to help users to locate relevant information on the web was already recognized in the mid-nineties. Their findings, based on six months use of a virtual assistant, were very promising, but the team acknowledged that further work was required, including addressing the issue of the virtual assistant's facial expressions.
An avatar can represent a tutor or an instructor who will guide learners through the materials [8]. A study carried out by Mazlan and Bird [9] investigated whether the use of avatars in online learning settings could motivate students. The results of the study indicated that the "students expressed positive interest in avatars and their motivation to learn". It would be interesting to investigate the effect of changing the role of the avatar; for example, would learning be improved if the avatar were a companion or co-learner rather than a tutor?
Smailes and Gannon-Leary [2] compared virtual support models for student mentoring and support purposes, to ascertain which would be best suited for their students' needs. Their study investigated the use of virtual learning environments (VLE), social networking sites and virtual worlds. Their findings were that a VLE 'lacks excitement', social networking was popular with students and virtual worlds were viewed as an unknown territory by many of their students. Based on this work, they carried out a pilot study on the use of social networking for student support in their business programmes. The study found that not all higher education students were familiar or even comfortable with virtual worlds, at least not in all subject areas or when used in an educational setting.
Use of avatars in health-related work has been considered in a number of studies. Kim and Sundar [8] investigated whether the use of avatars could improve a person's self-image. The findings indicated that for those with highly discrepant self-image, a virtual avatar becomes 'more of a motivating force' and could have a positive influence on their behaviour in the real world.
Similar research on self-image by Peña and Kim [10] focused on avatars as part of 'exergames' (video games for exercise; virtual tennis in this case). The study experimented with manipulating the participants' weight from slim to normal to obese to see if this had an effect on their physical performance as they played. They demonstrated that participants increased their physical output when both players' weights were kept normal, but play slowed down and was 'sluggish' when they were made obese. They concluded that "traditional psychological processes can be put to the service of increasing physical exercising and hopefully improving people's well-being through virtual experiences." The literature provides a number of different descriptions of what is understood by the term 'agent' or 'avatar'. For example, [11] states that "an agent is defined as an acting entity, which includes artificial intelligence that renders the control. An avatar, by contrast, is a virtual representation of a human being which is controlled completely by the human." We could describe an avatar as a computerized representation of a character representing a user in another environment. Avatars are used extensively in many computer games, where an avatar character may be controlled by the user.
Studies have been conducted on how people react to agents and avatars, in an attempt to establish why users have a social reaction towards them regardless of the knowledge that they are conversing with a machine. The authors [11] and [12] investigated the nature of the changes in people's communication when they interact with intelligent agents, as compared to human-to-human communication.
Cafaro et al [13] found that "in first encounters people quickly form impressions of each other's personality and interpersonal attitude." They found that even when one of the participants was a virtual agent, it took an average of only 12.5 seconds for subjects to form an impression. In other words, they reacted to the agent in a similar manner as they would to another human.
According to Baylor [14], the nature of the agent interface does not have much direct impact on actual learning, but does have a positive effect on learner motivation and self-efficacy. Baylor's study discusses the design of agents and the importance of the agent's visual appearance as "the most important design feature as it dictates the learner's perception of the agent as a virtual social model, in the Bandurian sense". Another key feature proposed is "message delivery through a human like voice with appropriate and relevant emotional expressions".
The Microsoft Office agent 'Clippit' was introduced in the 1990s [15], [3]. This character was intended to act as the users' personal assistant by speaking and acting on voice commands. It was generally seen as being too intrusive and rather 'characterless'. Although users' unfamiliarity with the technology at the time might have been a factor in this, it may be noted that a personal agent will fail if user expectations are not met.
Some rapid advances have been made in technology in recent years, but many challenges remain and one of these is the use of natural language with agents; this requires in-depth understanding of linguistics and semantics in order to create a truly intelligent system [3]. Siri, the voice recognition app and intelligent digital assistant from Apple, uses natural language to help users to find answers to their queries. While Siri's launch caused much excitement, this soon deteriorated when Siri's limited intelligence became apparent. When an avatar, as part an intelligent system, can engage in conversation with a human being it can be described as an Embodied Conversational Agent (ECA). Cassell [16] states that "Embodied conversational agent interfaces are specifically conversational in their behaviors and specifically humanlike in the way they use their bodies in conversation". As an example of an ECA, Cassell's creation, REA (Real Estate Agent) acts as a salesperson interacting with clients and showing or selling them virtual properties. REA can also engage in 'subtle human like conversation' with clients at appropriate times and use such phrases as 'I see' to indicate agreement with a client or that she is listening. These may appear to be simple features, but human conversation is extremely challenging to model in a computer system and this system required lot of sensors and computational resources to function [3].

Research methodology and analysis of the need for a VA
To establish the extent of the perceived need for a virtual assistant, we needed to gather quantitative data on the nature of students' interaction with existing support staff, and also to target a particular university service to obtain the opinions of members of support staff as to the potential value of a virtual assistant system. We chose to use a questionnaire to gather the student data, as it would have taken considerable time to gather data by interviews and the qualitative nature of such data might have skewed the perception gained. Furthermore, we wanted students to tell us both negative and positive aspects of the support they received and the anonymity of the survey made it more likely that we would obtain genuine responses, as compared with interviews or focus groups. For consistency and quantification, the questionnaire contained only closed questions. Aspects explored by the questionnaire included the frequency of contact, the nature of the students' queries, the modes of contact, i.e. face to face, online etc. and the quality of the students' interactions with services. We also sought students' opinions as to whether they would be likely to use a VA for out-of-hours assistance.
The survey was carried out with 125 students from the School of Science and Technology at Middlesex University. The questionnaire was given to students at foundation level (FY), first year (Y1), second year (Y2) and third year (Y3) undergraduate level. Some of the findings were as follows.
The survey showed that students need to contact their tutors most often at FY and least often in Y2. Most students found they needed to contact their tutors frequently throughout the year. The majority of tutors responded to student queries within two days. The main reason for students contacting tutors was for academic issues, although 30% of the Y1 students contacted their tutors for non-academic issues. The survey confirmed that students regularly accessed online support facilities.
Surprisingly, students had almost no interest in using contact via phone or social media, which contradicts the findings in [2]. Based on our survey, students currently prefer communicating with tutors face-to-face and by email.
A significant proportion of students at all levels stated that it was likely they would contact tutors outside teaching hours. There is a possibility that students have misinterpreted this question; we consider teaching hours to be normal working hours, whereas students may have interpreted this as time outside the classroom. However, the data did suggest that students value the opportunity to contact their tutors at any time.
The survey established that students require support on all aspect of their studies, from general administrative issues to pastoral and academic queries. Whilst students value face-to-face communication and support, they also now expect round the clock support [1].
Students responded very positively when asked how likely they would be to use an online tool such as a Virtual Support Assistant. Almost 80% of students responded that they would be (quite likely to very likely) to use such a tool. It appears that students value direct contact with tutors (either face to face or by email) for academic matters, but would also value the additional support provided by a VA. The university's employability support staff were also interviewed as to the potential value of a virtual assistant system, and responded positively.
There is evidence [17] that students prefer information provided via the spoken word to written information. Of course, the best way to achieve this is by person-toperson communication. Recorded audio can provide the personal touch but no body language, gestures etc. Recorded video provides the element of body language, but neither audio nor video recordings allow the possibility of interaction to guide the user towards the desired results. An avatar-based system does not have the personal touch, but provides another dimension beyond plain text and facilitates a personalized dialogue.

Virtual Assistant support system
The idea of the so-called 'chatterbot' came to prominence with Alan Turing's seminal paper of 1950 [18], in which he introduced the idea of the 'Imitation Game' test, now usually referred to as the Turing Test, to establish whether a machine could carry out a convincingly realistic dialogue with a human being. An early example of a chatterbot was Joseph Weizenbaum's Eliza in 1966 [19], which used natural language processing and pattern recognition techniques.
In the 1990s, second generation chatterbots appeared, using artificial neural networks. More recent systems often use Artificial Intelligence Markup Language (AIML), which is XML-based and uses tags to make the implementation of dialogues easier. This is what we used to develop our system's knowledge base. The VA was designed to be a web based application (Fig1) that can be accessed 24/7 by staff and students. The system was programmed to interact dynamically with the user and to provide answers to context-specific questions. The web site layout is based on an accordion style menu. The VA is in the form of an avatar that interacts with the user in both text and speech. • XAMPP to build the system on a localhost.
Character Hosting was used to create and customise the avatar, its voice and accent, and add animations. Program-O has its own AIML interpreter, which was used to interpret the rules for user interaction with the avatar. It is coded in PHP and has an inbuilt relational database that can be used to store the interaction between the user and avatar. These can then be retrieved and analysed by the programmer in order to improve the responses and also provide the user with a log of their conversation. AIML code was used to enter the rules for topic-specific questions from the user and provide the knowledge base for the responses.
As mentioned earlier, the university agency targeted was the employability service, and we worked closely with a key member of staff within the service to learn about the nature of the advisory role, how a virtual assistant might augment that role and thereby to develop the content for the AIML rules.
AIML represents dialogue as so-called categories, each comprising a user input pattern and the corresponding system response. The knowledge base comprises the set of all categories. The category tag defines a unit of dialogue, the pattern tag defines a possible user input and the template tag defines the corresponding system response. For example: <category> <pattern> Hi </pattern> <template> Hello I am your virtual assistant. What is your name? </template> </category> <category> <pattern>Hello</pattern> <template><srai>Hi</srai></template> </category> <category> <pattern>Good morning</pattern> <template><srai>Hi</srai></template> </category> ... The <srai> tag causes the system to provide the same response for the second and third categories as it does for the first one. This illustrates one of the main difficulties with constructing this kind of knowledge base; anticipating and designing appropriate responses for the user's potential inputs. This is made easier by the use of wildcards, as in the following example. <category> <pattern> My name is * </pattern> <template> <set name ="username" > Hi <star/> </set> </template> </category> Complex patterns may be mapped into simpler patterns by the following symbolic reduction technique. <category> <pattern> What is a CV? </pattern> <template> CV stands for curriculum vitae, a record of one's work and educational history </template> </category> <category> <pattern> What is a JD? </pattern> <template> JD stands for job description, to which you should try to match your own skills </template> </category> <category> <pattern> Do you know what a * is? </pattern> <template> <srai> What is a <star/> </srai> </template> </category> Synonyms may be resolved as follows. <category> <pattern> CV </pattern> <template> A record of one's work and educational history </template> </category> <category> <pattern> Resumé </pattern> <template> <srai> CV </srai> </template> </category> The above are examples of the techniques needed to construct a reasonable knowledge base for the system. The content was developed in collaboration with the university employability service. Figure 2 shows a typical dialogue

Evaluation
To evaluate the pilot system we once again turned to a questionnaire, which was completed by 10 final year students after having being given the opportunity to use the system. The rationale for using final year students for the prototype evaluation was that the system was implemented within the limited domain of student employability, which was of particular interest to those in their final year. However, on reflection, a larger sample of students would have been preferable and consequently our results can only be viewed as indicative at this stage. The questionnaire comprised mainly closed questions, but students were also invited to add additional comments. Not all students answered all of the questions.
The survey revealed that 50% of the students found the avatar system either extremely easy or moderately easy to use; none of them found it difficult to use.
89% of the students found the avatar's responses to their questions very relevant, quite relevant or moderately relevant. This was encouraging, as the programming of the rules to produce a meaningful dialogue with the avatar was one of the most difficult aspects of the project. Despite the use of wildcards and pattern matching, and the limited nature of the domain, it was still difficult to anticipate and program meaningful reactions to all possible user queries.
78% of the students found the avatar's tone of voice either very appealing, quite appealing or moderately appealing. This was unexpected, as we were not entirely satisfied with the quality of the avatar's voice and previous research [11], [12], [13] has found that subjects respond to artificial voices in the same way as they do with real human voices.
89% of the students found the use of spoken responses either very appealing, quite appealing or moderately appealing, when compared with text responses. This might have been due to the novelty of the approach, but nevertheless was encouraging.
The positive reaction of the students to the avatar system was exemplified by their responses to a question about the extent to which avatar-based interfaces should be used for interacting with the university's online services. Most of the students said that such systems should be used for all services, the majority of services or a lot of services, while one student thought they should be used sparingly. No students believed that avatars should never be used.
The students' additional comments provided some insights. They gave us some useful feedback on the general layout of the interface and additional employability-related subject content that they would like to see included in the system. Some of their comments (in italics) on the nature of their interaction with the avatar were: "Though it had limited responses, it gave useful information." "I like that the avatar reads the text to you, this makes the user fully understand and not miss certain words nor paragraphs. i also tried making a couple speeling mistakes such as adding extra letters to words in the question input tab, the system still provided rough the answer and/or rough guidence." (sic) "The system was relatively consistent and user friendly." These were encouraging, as they indicated that students could have a meaningful and valuable interaction with the system. Some responses were contradictory, for example: "…quick response to question…" but also "I would like the avatar response to be quicker than it is now." "… make the avatar more appealing, it looks bit dull at the moment" Although only one student made this point, it is generally accepted that it is challenging to make avatars express facial expressions appropriate to the words being spoken, which would be the best way to make them more appealing [ 16], [20]. "The avatar system is extremely user friendly and this is the very first time I am using it. Receiving a spoken response gives you a feeling that someone is physically there to help you." This last response was very interesting; our intention with this system was to try to simulate the supportive dialogue that a student might have with a member of staff, and this comment suggests that we might, at least in part, have gone some way towards achieving this.
A member of the employability service also evaluated the working prototype using a 'think aloud' protocol, which was recorded and later transcribed. One outcome from this dialogue was input from the employability team on the integration of the VA avatar into the employability service website. As a result, the home page was redesigned, with the avatar sited at the top within a clean and simple design, with hyperlinks related to the different types of support offered by the employability service. There were also links to other relevant sites such as the university's student job advertising site and a selfassessment tool for students seeking jobs. Students struggling to find what they needed could then immediately seek help from the VA.
Another outcome of the evaluation was the identification of some minor issues with respect to the content provided by the VA; the employability team representative pointed these out and the appropriate amendments were carried out.
Additionally, based on the evaluation, some of the chunks of information provided by the avatar were considered by the employability representative to be too long for aural delivery. This was a highly relevant observation, which went to the heart of the nature and role of avatar-based interfaces. Clearly, a major advantage of the avatar is the ability to guide the user towards solutions through the process of interaction, and conducting this dialogue aurally helps to give something of the feel of a human-to-human conversation. However, where there is a large amount of information to be conveyed, the aural method is not appropriate; nobody likes to listen to an extended monologue, and it is easier to explore a large amount of information when it is in text form. This enables the material to be scanned in any order, and one's focus to be concentrated on particular areas as necessary.
The avatar's aural delivery is linear in nature, and precludes such flexibility. This emphasized that the avatar was most useful when used as a tool to aid navigation through material to locate relevant information, but that it was only appropriate for actual delivery of that information where this was in small chunks. For larger chunks of target information, the avatar should help the user to locate these, after which they would be consumed in text form.
As a result of this evaluation, the role of the avatar was more clearly defined, as an engine to advise students by guiding them towards appropriate resources, such as an example CV, samples of typical interview questions, lists of recommended job sites or booking an appointment with an advisor. Following these changes, the employability service confirmed that they were satisfied the system was fit for purpose.

Conclusions
The introduction of a virtual assistant to supplement facetoface advice was considered to be desirable by a significant sample of undergraduate students at every level and also by members of the university's employability service.
We successfully developed, in collaboration with the employability service, a virtual assistant using a rule-based engine and a text-to-voice avatar front end, to advise students on all aspects of employability. Feedback from students' evaluation of the system was generally positive, with some indication that our goal of providing an (albeit limited) out-of-hours version of the support provided by university employability service staff had been achieved. The employability service evaluation of the prototype was positive and they foresaw using the VA in providing 24/7 student support for their service.
This technology is now to be used to augment other university support services and also some aspects of the role of academic tutors. Ultimately, the success of the project will depend heavily upon the quality of the content and the sophistication of the interaction rules. Obviously, such systems are unlikely to pass the Turing test, but it is important that they are able to provide responses relevant to the user's questions, to maintain a reasonably convincing dialogue and to guide the student towards the required information, and this pilot study has shown that, with careful design, this is possible. 7 In