E-learning project assessment : A new approach through the analysis of learners ’ posts on social media

E-learning assessment is a key aspect in the overall e-learning process. There are several parameters to consider during the assessment. In recent years, several sets of factors, called Critical Success Factors, have been defined to provide a structural approach to assessment. They focus on many aspects but, in our view, they do not properly consider student satisfaction with courses. In e-learning applications, student opinion must be examined where it is expressed: on e-learning course social pages and/or social pages outside the platform but specific to the e-learning course. The problem is that these resources are unstructured and thus it is important to structure these resources before using them for assessment. In this paper, we discuss a proposal that can capture student opinion from social pages, combining several techniques, such as Natural Language Processing, Information Extraction; ontologies that help us to understand what and how students discuss about e-learning courses.


E-learning
is one of the newer learning trends which challenge the traditional "bucket theory" or "banking" concept of education [2].If the economic assessment of elearning is quite simple, it is much more difficult to evaluate its efficacy because there are many parameters that must be considered.The evaluation of an e-learning project is still an unresolved problem.
As described in the related work section, in the literature there are several approaches, each one based on specific methods that try and capture specific factors which can characterise the quality of the entire project.
The simpler approaches evaluate the obvious aspects of the courses communicated through the e-learning environment, such as their structure and content.Thus, in this case, a standard evaluation has aspects, such as "general information and accessibility" and "organisation" (in which the learners are informed correctly about the course and its objectives and procedures); "language" and "course content" (managing the core elements of the course); "layout" and "learning resources" (the quality of the layout and interactive features of the e-learning environment, and the accessibility and accuracy of learning resources); "goals and objectives" and "evaluation".In addition to these parameters, overall aspects which relate to evidence that the course has been piloted successfully and that the learners can achieve the objectives of the course is very important.
Other approaches try and standardise the measured features, defining a set of specific factors called Critical Success Factors (CSF) [3] that are strictly connected with the success of the course [4][5] [6].Other researchers have examined the connection between perceptions of information communication technologies (ICT) used in the classroom and students' quality of learning [7][8] [9][10].This kind of evaluation is based on the idea that understanding the perception of students regarding the effectiveness of technology use and their proficiency in, and knowledge of, specific types of ICT tools are both influential on, and critical to, the success or failure of the integration of ICT and, thus, of e-learning environments in standard education settings.
In many cases, internal and external reviewers provide an assessment of the courses but the evaluation is strictly related to their experience and their expertise.On this basis, there are several aspects that influence the e-learning assessment; they can be individual (such as physical characteristics, learning history, learner attitude and motivation, and so on), environmental (organisation of the course, availability of course materials, and so on), or contextual.Other aspects are related to the platform's capabilities and its actual usage in a real working environment, analysing the learners' behaviours and results.
In our view, these aspects are not comprehensive because they consider the evaluation approach by types of data used but do not properly consider the opinions of the participants.
Considering that e-learning systems are defined and developed according to the specific needs of learners, it is clear that their success is closely related to the satisfaction of the participants.We think that, in the overall evaluation approach, the opinion of the participants must be taken into account.The problem is how to acquire this opinion.
In an e-learning system, where the course is entirely online, student satisfaction must be searched for where it is expressed, i.e., e-learning course social pages related to the courses (for example social pages within the platform) and/or social pages outside the platform but specific to the e-learning course.In this paper, we propose a first step towards the evaluation of e-learning project based on the learners' topics of discussion as posted on these social pages.The evaluation is oriented to the definition of student satisfaction, starting from student discussions on social media.
Our idea is based on the possibility of eliciting student perception of specific aspects of e-learning courses through the combination of several techniques and tools, such as text analysis, semantic web tools, Sentiment Analysis and so on.The output of this social media evaluation will be combined with other specific aspects of the overall evaluation of elearning.To develop this idea, we use a software tool that is able to identify any positive or negative trends, analysing the posts that learners leave in the forum or wiki of an e-learning portal.
In the following, Section 2 describes the related works regarding the approaches and the assessment methodologies defined in the literature.Section 3 illustrates the idea we propose and Section 4 illustrates an overall architecture which is useful in implementing the proposed idea.Finally, in Section 5, we draw some conclusions.

Related work
The evaluation of an e-learning project is an unresolved problem.As described in the introduction, there are several approaches that try and formalise a set of measurable parameters that can capture quality from inside the system.Several approaches are focused on the features of the platforms and others are focused on the use of the platforms.

Critical Success Factors
The first kind of approaches are based on the concept of CSF, i.e., "those things that must be done if a company is to be successful" [4].It is possible to apply the idea of CSF to the e-learning area.Papp [5] proposed a set of CSF for faculties and universities, considering elements such as intellectual property, suitability of the course for the e-learning environment, building of the e-learning project, and elearning project content.
Dillon and Guawardena [11] and Leidner and Jarvenpaa [12] define three main variables that affect the effectiveness of e-learning environments: technology, instructor characteristics, and student characteristics.Govindasamy [13] focused on the pedagogical aspects and defined seven CSF, including institutional support, teaching and learning, course development, student support, etc. Baylor and Ritchie [14] studied the impact of seven independent factors related to educational technology (planning, leadership, curriculum alignment, professional development, technology use, instructor openness to change, and instructor computer use outside school) on the dependent measures (instructor's technology competency, instructor's technology integration, instructor morale, impact on student content acquisition, and higher order thinking skills acquisition).
On the basis of a survey submitted to 47 students, Volery and Lord [6] identified three groups of CSF: technology (ease of access, interface design and level of interaction); instructor (attitudes towards students, instructor technical competence and classroom interaction), and previous use of technology from the student's perspective.Soong et al. [15] propose the following e-learning CSF: human factors, technical competency of both instructor and student, level of collaboration, and perceived information technology infrastructure.
On the basis of these CSF analytics, Benigno and Trentin [16] proposed an evaluation method based on the e-learners' performance.They confirmed the importance of positioning the students at the centre of evaluation methods because they are the main stakeholders of the e-learning platform.Their CSF are student characteristics, effective support, student to student interaction, learning environment, learning materials, and information technology.On the basis of the work of Benigno and Trentin, Erlin et al. [17] considered the role of asynchronous discussions in computer-supported collaborative learning (CSCL) tools.These discussions make them actively engaged in sharing information and perspectives by interacting with other students.Even basic CSCL tools enable the development of these threads where the learners can access text, revise it or reinterpret it, which allows them to connect, build, and refine ideas, along with stimulating deeper reflection [18].There could be thousands of messages generated in a few months within these forums, containing long discussion threads bearing many interactions between students.
Focusing on the role of the student and considering learner satisfaction is one of the most important CSF to evaluate the effectiveness of an e-learning project and, combining this with the significant number of discussions and posts by E-learning project assessment: A new approach through the analysis of learners' posts on social media 3 students in the e-learning environment, we propose the first step in a new evaluation approach.

Analysis of learners' posts on social media
Our approach considers the learner at the centre of a complex system and bases the evaluation of an e-learning project on the analysis of the strengths and weaknesses that the learners post on Web 2.0 pages related to the course.The approach is based on the idea that it is possible, on a statistical basis, to elicit student perception of specific aspects of an elearning course.If a large number of students criticises or appreciates a specific aspect of the course, it is probably true and, thus, this aspect must be taken in account.Thus, the focus of evaluation is moved from the course environment to the effect of the course on the learners and their perception of it.
Venkatesh et al. [19] underline the importance of understanding learners' perceptions regarding the effectiveness of ICT use.While Mokhtar et al. remark upon the central role of blogs that can be used with different aims in mind, such as student reflection [20] or to provide an alternative forum for regular classroom dialogue [21], the jury is still out regarding their usefulness.
The idea of using social networking to improve the quality of e-learning is not new.Kadry and El Fadl [22] explore the role of Web 2.0 technologies in learning and their influence on learners' behaviours, while Mahmood et al. [23] study how e-learning systems can improve performance using social network features.Our point of view is the following: we want to use these concepts to evaluate the courses and thus to understand where there are problems.
Considering Web 2.0 technologies, the ubiquitous nature of online social networks is highly relevant for our study.It was found that the group (in collaborative projects), the network (in discussions and queries) and the collective (in data-mining) play distinct roles whenever social software is employed for e-learning [24].Furthermore, Google, Facebook and Twitter are recognised as enabling students to learn outside of the classroom and build communities at the same time [25].We want to use learners' posts in blogs and social networks to capture user opinion about e-learning.
Starting from the previous assumption, in order to put in the idea into practice, it is important to consider at least two main aspects: topic detection (in order to understand the topic of conversation starting from the posts on social media), and evaluation of the contents of a topic through Sentiment Analysis (in order to understand the "mood" of the natural language).These aspects must be considered in the e-learning environment.

Topic Detection
Topic detection allows us to define the topics of dialogues which focus on the topics related to the courses and eliminating topics not relevant for the evaluation.
In the topic detection field, Cataldi et al. [26] underlines the role of Twitter and provides a new method of extracting emerging topics by analysing, in real-time, the emerging terms expressed by the community considering the time interval as a relevant factor.In [27], the authors investigate methods for exploiting community feedback in order to automatically identify high quality content.Agichtein's approach uses Yahoo!Answers as a case study; in fact, community Q&A is a popular information-seeking paradigm that has already entered the mainstream and it is significant for the kind of learner posts that we seek.Agichtein has also introduced a classification framework for quality estimation in social media.

Sentiment Analysis
Another important aspect in the evaluation of social posts is Sentiment Analysis.The purpose of the Sentiment Analysis is to recognise in a document the attitude (positive, negative or neutral) that the creator of the document has towards the object treated within it.The research topics in Sentiment Analysis range from computer science to linguistics, touching all aspects of Natural Language Processing (NLP); it is characterised by great complexity and this has led to relatively few studies being carried out in this subject [28].
However, research into Sentiment Analysis now demonstrates a new emphasis; works published over the last decade are many.In [29], the authors propose an extensive overview of all techniques for the detection of opinions and feelings within text documents.In [30], the authors warn of the spam problem, illustrating a case study on a set of reviews taken from Amazon, where the authors attempt to determine which are true and which are false (spam detection).[31] focuses on a set of reviews of films; the authors run experiments in Opinion Mining using Machine Learning techniques.In [32], the authors give the basis for the classification of text documents.Even space-time is an important factor in the process of Opinion Mining.In [33], the authors propose a model of queries of social networks in which they are integrated aspects related to space and time in the classical methodology of research.In [34], the authors attempt to determine the political orientation of the users [35], trying to determine the opinion expressed by Tweets through the use of supervised learning algorithms associated with the detection of emoticons; the results are encouraging with an accuracy of 81.3% (Naive Bayes algorithm), 80.5% (Maximum Entropy algorithm), and 82% (Support Vector Machine algorithm).Very similar results are found in [36].It is possible to see that the best results are obtained with a method of classification of unigrams type, considering only one term at a time during the classification (see [35] and [36]).In e-learning, Sentiment Analysis allows us to understand the perception of learners.A limitation is that the technique is very useful for English texts but they are not useful for other languages.
In the next section, we explain our e-learning assessment idea, starting from the related works discussed in this section.

How to support e-learning evaluation
The goal of the paper is to present a potential method of supporting e-learning project evaluation, using the unstructured data present in social media related to the elearning course to analyse the perception of learners as an important variable in the assessment process.In order to evaluate the opinion of the learners, it is important to analyse data that learners leave in the web, specifically, in the social pages (internal or external to the online web portal).The social pages we refer to are related to the e-learning project, where students can share documents and course material, along with exchange of signed or anonymous opinions, suggestions and feedback, which are useful for the evaluation of e-learning.This can be done through Web 2.0 tools, such as wikis, blogs and forums, internal or external to the elearning portal, in which students are invited to post their comments.
It is clear that these comments are not structured, written in informal way.Learners can decide whether to write anonymous posts in order to ensure the spontaneity of the posted comments about the quality of teachers, course content, teaching material, etc.In effect, a signed post could inhibit students from being completely honest and, therefore, would not provide a useful support for the evaluation of elearning.
The data from social media related to an e-learning course is very useful for evaluation but are, obviously, unstructured.Our idea is that, by collecting the unstructured sources contained in the portal, making them structured and extracting both obvious and hidden information, it is possible to obtain true learner perceptions of an e-learning project.As a result, improving e-learning projects by using this information will put the key stakeholders, the learners, at the centre of attention.
It is clear that the acquired information will allow improvements in course organisation.In general, on the web, learners express their opinions about something in a freeform way and without any constraints.As stated in [37] "User Generated Content (UGC) exchanged via large social networks is considered a very important knowledge source about all aspects of social engagements (e.g., interests, events, personal information, personal preferences, social experience, skills, etc.).However, this data is inherently unstructured or semi-structured".
The analysis of the e-learning social pages, the selection of a source useful to the evaluation process and the transformation from unstructured to structured, all requires several techniques, from NLP to Information Extraction (IE), and so on.The most important thing when extracting information from an unstructured source is to thoroughly understand the application domain and thus the e-learning domain.In order to do so, it is very helpful to use ontologies as a means to acquire the knowledge useful to analysing unstructured sources.There are many definitions of the concept of "ontology" in the philosophical field but one that is perhaps most suitable to this research is the definition given by Gruber in 1993: "ontology is a formal, explicit specification of a shared conceptualization."The term "conceptualisation" is the abstraction of a concept through the definition of its peculiar characteristics; the term "explicit" is connected to the fact that the constraints that contribute to the precise definition of the concept have to be expressed in a formal way.Finally, the term "formal" points out that an ontology must be defined through a formal language.Thus, ontologies constitute well-defined knowledge about a specific domain (in our case, the learning domain) which is helpful in the analysis and interpretation of unstructured sources.
The use of ontology in this work allows us to develop a better understanding of what students say about specific elearning topics.In order to choose the most appropriate ontology which best suits our purposes, we took advantage of the experience that we have had with other ontologies and domains.This experience has led us to use and perform empirical evaluations regarding ontology data completeness and accuracy and ontology data linkage.
The first estimation indicates how the concepts present in the ontology are complete and accurate in terms of coverage of the e-learning domain information.The second evaluation measures how the ontology concepts are related to each other, a high linkage is significant of a high correlation.The above evaluations are performed empirically by ontology and domain experts.Once the e-learning ontology is selected, it is possible to confirm, through reasoning tools and rule engines, that the ontology integrity constraints are not violated and that the ontology does not contain any formal error.
The unstructured sources are collected from social media and analysed in order to extract the information (words or groups of correlated words) which recurs the most and, therefore, is the most discussed by students.This is realised with tools that use the established methods and techniques of NLP and IE.The use of ontology simplifies the comparison of the information extracted from social media with the information from the specific e-learning domain.Indeed, the extracted concepts are compared with those contained in the ontology.The concepts that differ from those in the e-learning domain are excluded from the results because they could not provide any added value to the evaluation of the project.
The use of ontology is very helpful in precisely selecting only the information related to the specific application domain (e-learning).The ontology, in our proposal, is very useful to outlining a well-defined boundary that allows us to keep track of the learners' opinions which are useful to the assessment.
The problem at this point is to choose an ontology that can reflect the actual knowledge of the e-learning domain.Since there are several ontologies about this domain available online it is useful to reuse existing ontologies or existing concepts that are most suitable for the specific e-learning project and for the purpose of evaluation.As stated in [37] "Ontology reuse is an important concept in Ontology Engineering".According to [38], "it increases the quality of the application, achieves interoperability, improves cost in ontology development and helps applications agree on the domain concepts".
Another tool that helps in the analysis of unstructured sources is Sentiment Analysis.This tool is very useful because it helps us to understand the mood, that is, the meaning (positive, negative or neutral) of the specific post (or set of posts) in social media.While ontologies help us to understand what students say about a specific aspect of the elearning course, Sentiment Analysis helps us to understand how students talk about a specific aspect of the e-learning course.In order to clarify our idea we can summarise it in the following steps:

EAI
• Identify the source of information: in the e-learning context, the unstructured data are in the e-learning platform and/or in the specific social media to which learners have access.It is important to know where sources are available.• Identify a domain ontology in order to facilitate IE, to filter out the information which is not associated with the specific e-learning scenario.It is possible, depending on the specific application domain, to integrate the selected ontology with some concepts.• Apply Sentiment Analysis techniques to understand the mood of the posts.
The outputs of the system will be: • Lists of those items which are most discussed by the users that pertain to the e-learning projects: these items will be organised in a way that is as clear as possible to the reader.• Information about how students talk about these items (Sentiment Analysis).
The list of items, in relation to the mood of the posts obtained through Sentiment Analysis techniques, provides an overview of the overall impression that learners have about elearning courses.The output will be analysed by experts and will be put into relationship with other evaluation aspects, to have an overall evaluation of the course.
The idea of structuring the unstructured information, and of extracting meaningful content from it, is not new: we have developed this idea using a tool established for the agri-food sector and presented in [39] and [40].The system is able to analyse and extract information related to specific agri-food products from the users' experiences posted on blogs and social networks.The obtained information can be used for many purposes, such as marketing information (i.e., "which product is more interesting for the customer?")or commercial information (i.e., "what geographical zone has more interest in the product?").
The added value of this idea is its application to a completely different domain (e-learning) and the combination of the accrued experience with the need to understand learners' perceptions of e-learning and thus consider a specific aspect of the overall assessment.

The development of the idea
As mentioned in the introduction, the aim of this work is to evaluate learner satisfaction with an e-learning course by analysing the student posts published on the social web pages related to the e-learning course.In order to understand the sustainability of the idea, we designed a system to extract the key concepts of discussion from social media and understand if the opinions expressed by students are positive or negative.The architecture of the proposed system is shown in Figure 1.E-learning project assessment: A new approach through the analysis of learners' posts on social media • Domain Knowledge Filter: this is the component that interacts with the e-learning ontology and facilitates the comparison between the information extracted from the social pages and the information from the e-learning domain, by filtering out the inappropriate information.• Sentiment Analyser: this is the component in which the information retrieved and filtered by the Information Extraction and the Domain Knowledge Filter respectively, is analysed, with reference to its context in the text from which it has been extracted, in order to identify the positive, negative or neutral sentiments.The component is realised by integrating existing third-party tools that use Sentiment Analysis algorithms to look for words that carry a positive or negative connotation, understanding negations (i.e., "this e-learning course is interesting" vs "this e-learning course is not interesting") and modifiers (i.e., "this e-learning course is interesting" vs "this e-learning course is really interesting").• Information Database: this is the database that contains the main concepts extracted from one or more social web pages selected by the user for evaluation purposes.For each concept, this component stores information about the web URLs of the unstructured resources from which it has been extracted, its occurrence in each social page and the positive, negative or neutral sentiment.• Information Presentation: this is the component responsible for displaying to the user the social page elaboration results, in the form of a tag cloud and/or tables.In the first case, each concept is represented by a font of a size proportional to the number of occurrences identified in the text; the higher the number of occurrences, the larger the size of the font.The different colours suggest the sentiment level, which could be positive, negative or neutral.In the tabular representation, the component provides detailed information on a concept selected by the user from those displayed in the tag cloud.In addition to the indication of the number of occurrences in the text and the sentiment, it shows the unstructured web pages in which the concept is contained and the other concepts present on the same web pages The ontology used by the Domain Knowledge Filter component has been selected following a search of e-learning ontologies available on the web.This research was carried out by means of the ontology search engine Swoogle (http://swoogle.umbc.edu), a search engine for the semantic web that crawls the web for a special class of web documents, called Semantic Web Documents.As said by Sridevi and Umarani [41], even though there are a number of semantic web search engines, Swoogle is rated as best, because it ranks the ontologies using an adaptation of the Google Page Rank scoring method.
We have selected ontologies that best meet the requirements of our work.Among those ontologies, we quote the User Model Ontology (available at http://jelenajovanovic.net/ontologies/loco/user-model.rdf) for modelling users on an e-learning system, such as students, teachers, content authors, etc., and the EduOnto ontology for the educational assessment of the issues of learning and system evaluation that includes five different interpretative dimensions regarding teaching science (epistemological, referential, methodological, phenomenological and axiological) [42].Not all the ontology concepts are necessarily used by the system but there are those that match with those extracted from the students' posts on the social web pages.From the functional point of view, the social web page content elaboration starts from the selection, by the domain expert, of one or more unstructured social pages, followed by an indication of the degree of navigation depth among the selected pages (Figure 2).The system analyses the content of the social pages and extracts the most relevant keywords, entities and concepts, i.e., those having a higher number of occurrences in the text.This is done through the adoption of third-party tools that, in turn, make use of statistical algorithms and NLP technologies in order to analyse data.The extracted concepts are compared with those contained in the ontology and the concepts that are outside the boundaries of the e-learning domain are excluded from the elaboration output.The system also calculates the positive, negative or neutral sentiment of the information extracted.At the end of the data elaboration phase, the system displays the results in two different representations: • A tag cloud (Figure 3), a visual representation of the most relevant information in relation to the processed pages and the ontology concepts.The relevance of a concept is estimated according to the total occurrence within all the analysed web sources and the relative occurrence within each examined document.The information is displayed with a different colour (green, red or blue) in accordance with the identified sentiment (positive, negative or neutral, respectively).The method and the system described in the previous sections will be used in two case studies that aim to evaluate the customer satisfaction of two courses of the Faculty of Economics.The first is a course of the bachelor degree, most in e-learning mode, while the second is a master degree's course with some lectures in e-learning mode.
Both the courses will be held in the second period of the current academic year.

Conclusions
Evaluating the effectiveness of e-learning projects and the achievement of their goals and objectives is good practice, to understand the advantages and disadvantages, along with the strengths and weaknesses of an e-learning project.Thus, many sets of CSF, such as contents, technologies, etc., are defined but, in our opinion, these factors do not consider the most important aspect of the e-learning evaluation: the students' points of view.In order to capture and evaluate this aspect, it is important to analyse the students' comments and the discussions posted in the e-learning platform and on other social pages related to the e-learning project.The problem is that these resources are unstructured and it is important to make them structured before performing the analysis.
This paper proposes an idea useful in analysing unstructured resources, to extract important information about specific topics, and provide, as an output, a schema of the main topics discussed in social media with the "mood" of the students about specific topics.In this proposal, we use IE and Sentiment Analysis techniques and we use ontologies to filter out information unrelated to the e-learning domain.The paper also proposes an architecture which is useful in implementing the idea; the architecture analyses and extracts information related to the e-learning domain from the users' experiences posted on blogs, wikis, social networks, etc.Because of the freeform characteristics of Web 2.0 pages, the system interacts with an e-learning ontology in order to remove information unrelated to the specific domain.
As a result, the system provides a mechanism to structure the unstructured information extracted from the web and to support the e-learning domain expert in identifying the strengths and weaknesses of e-learning projects.We have already developed a similar architecture for the agri-food sector but we plan to develop the same architecture for the elearning domain where the problems faced are different.

Figure 1 :
Figure 1: System architecture It consists of the following components: • HTTP Request Handler: this component is responsible for the interaction between the expert user and the core module of the architecture.It queries the Social Pages Database, which stores unstructured web resources, in order to read and display to the user a list of social web pages related to the e-learning course, on which to perform the elaboration.The component receives a request from the user in terms of the unstructured resources to analyse.The request is forwarded to the Information Discovery and Sentiment Analyser macro module.The latter replies, through the Information Presentation component, to the request of the Request Handler with the elaboration results, which can then be evaluated by the expert user.• Social Pages Database is the database in which the web URLs of the social pages are stored.It can be powered by the web URLs of other unstructured resources identified by the expert user and related to the e-learning project.This component is queried by the HTTP Request Handler in order to retrieve the list of social web pages or to insert new ones.• Information Extraction (with NLP features): this is achieved by integrating existing third-party tools for content extraction from the social web pages through NLP features along with IE by sentence segmentation, entity detection and relation detection.The component interacts with the Information Database to store the extracted information, with the Domain Knowledge Filter component to communicate with the e-learning ontology, and with the Sentiment Analyser component for the identification of the extracted information sentiment.

Figure 2 :
Figure 2: Web source selection and insert form within the tool web page

Figure 3 :
Figure 3: e-learning project tag cloud within the tool's web page on e-Learning 03 -04 2016 | Volume 3 | Issue 10 | e E-learning project assessment: A new approach through the analysis of learners' posts on social media