Students ’ acceptance of peer review in Computer Science course

Peer review technique used in educational context could be beneficial for students from several points of view. Besides of developing students’ writing skills, critical thinking, practising articulation of own knowledge to the others and giving them feedback, it can encourage collaborative learning and boost the students’ interest in the course. In our web design course we successfully introduced peer review activities more than 2 years ago. In this paper we discuss the students’ acceptance of peer review applied on evaluation of other students’ projects. Received on 30 November, 2015; accepted on 23 March, 2016; published on 11 April, 2016


Introduction
Peer reviewing is widespread in many professional fields, e.g.science, engineering, health care, etc.In this process a paper, a software, or any other work outcome is assessed by a specialist who is of equivalent expertise level to the author.The goal of peer review is not only to evaluate the work but also to suggest how to improve its quality by adjusting and supplementing the content and the form.
According to the experience reported by several publications [1][2][3], the peer review approach can also be useful in the educational context.The outcomes of published studies confirm that thanks to the reciprocal evaluation of the colleagues' work the students could better develop their communication skills, critical and analytical thinking, constructive criticism, social learning etc.Additionally, this activity could also boost the students' interest in the course.
Therefore, more than two years ago we started to employ this activity into our course aiming to bolster up students' motivation for learning, course activities and for the course subject itself.At first, peer reviews were integrated with blogging which was also part of the course activities [4] -students evaluated the blog articles of their peers.Based on our previous research results [4][5][6][7] we can confirm that students' engagement in blog-based activities as well as their study results significantly improved after peer review was introduced.Despite the extensive involvement and better grading several students expressed their dislike for these activities, especially for blogging.Since we rated peer reviewing to be particularly beneficial for students, we adjusted the course activities next academic year and let students to peer-review their projects.As presented in this paper, after this change the preliminary results for students' acceptance of peer review are satisfactory.

Related work
The peer review technique was successfully employed into education in wide variety of different university disciplines (e.g. in Zoology, Information Systems, Environments and Engineering [8], Business Administration [9], etc. ) and subjects (mathematics [10], microeconomics [11], foreign language [12,13], etc.).Topping [14] offers a comprehensive review of 31 studies dealing with peer review integrated with educational activities in several other subject areas, e.g.art and design, engineering, social science, and other.
The most common way of using peer review in course activities is the mutual students evaluation of writing (essays, technical writing or blog articles) [11,12,15] or oral presentations [9,16].
Gehringer [17][18][19] successfully used peer review also in other educational activities such as annotating lecture notes, collecting sources related to the course topic, creating test questions, making up a problem on some topic, etc.In some assignments, after initial feedback phase students are enabled to correct the errors pointed out by reviewers.Besides the reviewing others work students also evaluate the usefulness of the reviews they got from colleagues to their own work.
Many authors not only report the incorporating of peer review into their courses and its acceptance by students but also explore the potential of this technique in student's assessment and the influence of peer review activities on student's learning [4][5][6].As stated in several studies [11,20], students tend to give better rating to their peers than the teacher gives.In spite of this fact, it is possible to identify weaker works based on the students' peer reviews [20].MacAlpine [21] and Gehringer [18], who deal with the format of peer review, point out the importance of prescribed peer review structure and good specification of all criteria and aspects to be assessed.
The peer review technique was also used to determine the relative contribution of each group member working on a group project assignment [22].

Course description
Our research was conducted on a web design course that is obligatory part of master study program in Applied Informatics.Besides of masters' students many bachelor students voluntarily enroll in this course every year.
The course focuses on front-end web design issues including both desktop and mobile websites and concentrates predominantly on web design methodology (such as prototyping, usability testing, and user centered design) and web quality standards (accessibility and usability, content quality standards, etc.).
Neither lectures nor practicals comprised in the course are mandatory.The main course activity is a project -a full semester assignment that constitutes one of the essential parts of course evaluation.Apart from the project the course evaluation includes written exams (midterm and final) and extra points for activity during practicals.The best grading students can get for the points earned for all these activities is C (on the scale A, B, C, D, E, Fx, where Fx stands for a failure).The best students are allowed to improve their grading at an oral exam.
The main objective of the project is to practise the knowledge and skills the students are supposed to master during the course.Therefore the project assignment for every particular student is to develop her personal blog with typical blog features, both in desktop and mobile version, including sufficient amount of meaningful content.In the past runs of the course, the assignment was only evaluated as one final submission.As we observed that the students only started to work on the project a few days before the deadline, several years ago, we split the project into three phases, each of them evaluated independently.Thus the students had to spend more time working on the assignment what brought about better learning outcomes.For further improvement, in the last semester each of the phases was supplemented with a peer review round.

Peer review
In the course of each project phase students had to meet three deadlines (see Figure 1).Firstly they developed their web application according to the current phase requirements.Consecutively they submitted the project for the peer review in a predetermined deadline.Each student, who successfully submitted her assignment in the given phase, was assigned three randomly chosen submissions to review.The reviewing period took approximately 3-4 days (second deadline).After reviews were delivered, the authors were given few days for correcting their projects according to the peers' comments (hereafter, the project improvement phase).Only after this phase projects were submitted for teacher's evaluation (third deadline).Since the students were not skilled in evaluating others work, a structured review form with several questions was prepared to facilitate the reviewing process.The questions in the form were different in each of the three phases and were directed towards the goals of each phase.They more or less covered the evaluation criteria used by the instructors.The reviewer had to answer each question with a rating ranging from 1 (very poor) to 5 (excellent), and in addition had to provide a verbal justification for the given rating.The reviews were blind i.e., the reviewers knew the identity of the authors but not vice versa.In each phase, the teachers evaluated not only students projects but also the appropriateness of ratings and reviews the students gave to their colleagues.As reviewing was completely new activity for students, the teachers' evaluation of students reviews was not too strict in the first round.This way, the reviewing in the first project phase was intended more or less for training.More detailed information about this peer review methodology and project phases can be found in [23][24][25].

Research of students' acceptance
To find out the level of students' acceptance of peer review strategies used in the course, we prepared an anonymous questionnaire.Students were asked to complete it in the beginning of exam period.Although participation was optional, the questionnaire was completed by 54 students (93.10%).However, a few of them did not answer all questions.

Questionnaire description
Questionnaire consisted of 16 (semi)-closed questions with possibility to choose more than one option in most cases.Questions were divided into two sections: the first one was focused on students' opinions about peer review in their learning and the second one was oriented to the overall course evaluation.

Students' opinions about peer review
Initial question explored various benefits of reviews given to a student by her peers.As shown in Figure 2, nearly all students chose the option: to correct omitted deficiencies.However, more than a third of respondents admitted that the reviews received from their peers helped them to improve their own reviewing skills, and a quarter of them could now solve some issues that they were not able to solve before reviewing.Also, one quarter of students averred that their understanding of project assignment improved after they received reviews.
If student chose the option other, she should explain it.In some of these cases students appreciated the possibility to gain another opinion to their work: I got alternative view on my project; or viewing other projects inspired me in my work.However, there were also a comments like: I "corrected" stuffs that were correct, according to bad advice of my reviewers.Since we expected such situations, students were asked to specify their experience with incorrect advice.
At first we asked whether they corrected their projects before final submission according to reviewers' advice.Although there were five options, students chose only 3 of them or did not answer (see Figure 3 (a)).no answer Since we assumed that students would not correct all admonished shortcomings, we also explored their reasons to do so (see Figure 3 (b)).The most often selected option (I ignored bad advice from my peers -66.04%) indicates that the students are able to distinguish a right advice from wrong ones.15.09% of students claimed that they did not have enough time to make corrections and almost 10% of them admitted their laziness to work on projects.
Peer review process could also be beneficial to the reviewer.Therefore we asked students whether they gain some benefits while reviewing others' work.We expected the most popular option would be I could gain more points.However, although 61.11% of respondents chose it, there were even more students (68.52%) who stated that they realized shortcomings in their own projects thanks to the reviewing projects of their peers (see Figure 4).Also the fact that only 3.70% of students claimed that peer review was not beneficial to them can be taken for a good result.While many students welcomed the project improvement phase, there were also some who tended to abuse this new option.Reviews they got from their peers did not influence their final grade and therefore some of them had submitted an incomplete project intending to finish it during the reviewing phase.Accordingly to this, we asked students whether projects they reviewed were appropriately finished considering the respective project phase.As depicted in Figure 5 (a), 3.77% of students stated that all the projects and 81.13% of students stated that most of the projects they had reviewed were finished.

I did not review unfinished projects
In the next question, we explored how students perceived the unfinished projects.57.41% of respondents declared that they reviewed unfinished projects that were assembled during reviewing process and this did not cause any problem to them (see Figure 5 (b)).On the other hand, 16.67% of students considered reviewing unfinished projects as a difficult task.
The last question focused on the perceived benefits of the project improvement phase.Definitely the most popular option was the first one: I had a chance to get more points for the project, since 88.89% of respondents chose it (see Figure 6).Students also appreciated extra time they got for final submission (61.11%) and a quarter of participants welcomed the project improvement phase since their reviewers helped them in eliminating mistakes.The fact, that options it was not beneficial to me and most of peers' advice were bad, so it did not help me were not chosen by any student, is very satisfying.

Students overall course evaluation
This part of our questionnaire comprised the same questions as those in the official student feedback form provided by faculty every semester (see Table 1).We used this strategy with the aim to reveal the differences between results of these two surveys, as discussed in our other paper [26].On the other hand, this part of survey helped us to better understand the students' attitude to the course and its activities.
A. Course quality.The first question -treated as the most important -was oriented on students overall rating of the course quality.Students answered by choosing a value from Likert scale 1-5 (depicted by stars), where 5 was the best.As depicted in the Figure 7 nearly 40% of students rated the course as good or very good (four or five stars).Together with those who chose three stars nearly 74% of students assigned the course at least moderate evaluation.This resulted in the average rating as high as 3.15.According to the objections to the activities of this course the students used to raise in previous years [4], this rating can signalize raising students satisfaction at the course.Students could choose one from the following options: incomprehensible, too difficult, just right, too easy, and trivial.The chart in the Figure 8 shows that most of the students (66.04%) considered the course to be properly adjusted -neither too easy nor too difficult.C. Course interestingness.The aim of the third question was to find out whether and to what extent the students were interested in the course topics.The options to choose were the same as in the first question -one to five stars.More than 58% of students rated our course interesting or very interesting (four or five stars).More detailed results are depicted in the Figure 9.
While exploring the answers to this question a curious fact was noticed: the students almost split into two groups -the one who found the course topics interesting or very interesting (the above mentioned more than 58%) and the other who found the course uninteresting or very uninteresting (nearly 36% who rated the course by one or two stars).Only less than 6% of all students assigned the course three stars.This could be caused by the fact that although this course is specialized, it is mandatory for all students of Applied Informatics.Therefore it could be uninteresting for those who did not specialize in this area.This issue was explored in more depth and compared with the outcomes from previous years in our other paper [27].The comparison showed the improved opinion of students and their higher satisfaction with the amount of course work this year.

E. Course attendance.
Here we asked the students how often they attended in-class teaching.There are two reasons why this question is included: the answers can illustrate how much the course is attractive for students, and how reliable are the other answers gained from this questionnaire.We believe that if there are more students who engage in in-class teaching regularly, the results in other questions will be more trustworthy.
The results are depicted in the Figure 11.There were less than 48% of students who attended in-class teaching more or less regularly (options 90-100% and 60-90%), which we consider to be quite alarming state.We hypothesized that this low attendance in in-class teaching could affect students' opinions explored in our survey.
For better understanding of relations between students' attendance and their answers to other questions, we calculated correlation coefficients for these variables.Unfortunately, all correlation coefficients we obtained were too close to zero to be significant.Therefore we can claim that there is no correlation and based on this we cannot describe any relation between students attendance and their opinions in other questions.We had two reasons for including this question into our questionnaire.The first reason was the above mentioned comparison of our results with the official student evaluation of teaching (see [26]).The other one was our assumption that the comprehensibility of the course requirements and rules could influence the overall student's evaluation of the course.
In this case, more than 72% of all students considered requirements and rules of the course to be clearly defined -they chose answers yes and definitely yes (see Figure 12).The remaining less than 28% answers no or definitely no relatively well corresponded to more than 27% of answers representing last two columns of the chart in Figure 11.These results seemed to prove the relation between the students' participation in inclass teaching and their understanding of the course requirements and rules but, as it was mentioned above, there was no correlation found.However, all rules and conditions were available at official course website and lecturer explained it at very first lecture, as well.G. Recommendation to the others.The very last question asked students to recommend or not to recommend the course to the others.The choice of the answers was the same as in the previous question: definitely yes, yes, no and definitely no.Counting positive answers of both types together and similarly counting negative answers of both types together resulted in a finding that nearly 58% of students would recommend this course to the others and about 42% of them would not (see Figure 13).Although the number of students who answered positively was higher, we consider the number of the others unsatisfactorily high.

Conclusions and future work
Peer review is an activity which, combined with an appropriate methodology and suitable educational context, can become a powerful tool bringing numerous benefits to students as well as to teachers.
In this paper we focused on peer review activities integrated into a web design university course.Despite the fact that our previous attempt to combine peer review with blogging activities was accompanied by higher engagement of students and significantly improved study outcomes as well, students were not satisfied with this activity.Therefore we used peer review in combination with project assignment in the last course run.The aim of this study was to find out the attitude of our students to this educational activity in the changed conditions.According to the results from the questionnaire conducted in the end of semester, the redesigned activity was positively accepted by students.They assumed it not as a task created for its own sake, but as an activity that brought added value in their learning.The improved students' opinion of this course can be demonstrated also by the results from the second part of our questionnaire.The overall rating of the course quality rose compared to the previous years.Most of the students considered the course difficulty appropriate and the course topics interesting.The required amount of work in the course was evaluated as proper or too much but also in this question the results improved comparing to the previous years.However, there was still rather high number of students who would not recommend this course to the others which we treat as a challenge to further improvement.
In our future teaching activities, peer review will be used in combination with group projects where group members will review each others' work.We believe that this approach can bring new benefits to the students and further improve students' attitude to our course this way.Moreover, it can also be beneficial to the teachers, providing them with the tool supporting more fair evaluation of individuals.

Figure 1 .
Figure 1.Illustration of the peer review process.

Figure 4 .
Figure 4. Do you think that you gain some benefits while reviewing others' work?A: I learned how to test web projects; B: I learned how the project assignment was perceived by my peers; C: I realized how many different types of mistakes can appear on websites; D: I realized shortcomings in my project during peer reviewing; E: I trained my verbal skills; F: I learned how to give constructive criticism; G: I could gain more points; H: nothing; I: other

Figure 6 .
Figure 6.Do you think that the chance to correct your project before final submission was beneficial to you? How? A: I had a chance to get more points for the project; B: peers gave me advice how to remove mistakes I was not able to solve; C: it helped me, but they gave me also bad advice and I lost points because of it; D: most of peers' advice were bad, it did not help me; E: I got extra time; F: it was not beneficial to me; G: I do not consider submitting projects this way as a fair option; H: other

Figure 7 .
Figure 7. Overall rating of the course quality.

Figure 8 . 5 EAI
Figure 8.How difficult was the course?A: incomprehensible; B: too difficult; C: just right; D: too easy; E: trivial.

Figure 9 .
Figure 9. Was the course interesting to you?

Figure 10 .
Figure 10.Was the amount of work appropriate considering the number of credits?A: absolute killer; B: too much; C: just right; D: too little; E: what work?.

Figure 11 .
Figure 11.Approximately what proportion of the in-class teaching (lectures, labs, etc.) have you attended?

Figure 12 .
Figure 12.Were the requirements and evaluation criteria of the course clearly defined?A: definitely yes; B: yes; C: no; D: definitely no.

Figure 13 .
Figure 13.Would you recommend this course to other students?A: definitely yes; B: yes; C: no; D: definitely no.

Table 1 .
Questions from official student feedback form.