Research Article
Reviewing the reviewers: A study of author perception on peer reviews in computer science
@INPROCEEDINGS{10.4108/icst.collaboratecom.2010.33, author={Conny K\'{y}hne and Klemens B\o{}hm and Jing Zhi Yue}, title={Reviewing the reviewers: A study of author perception on peer reviews in computer science}, proceedings={6th International ICST Conference on Collaborative Computing: Networking, Applications, Worksharing}, publisher={IEEE}, proceedings_a={COLLABORATECOM}, year={2011}, month={5}, keywords={Computer aided software engineering}, doi={10.4108/icst.collaboratecom.2010.33} }- Conny Kühne
Klemens Böhm
Jing Zhi Yue
Year: 2011
Reviewing the reviewers: A study of author perception on peer reviews in computer science
COLLABORATECOM
ICST
DOI: 10.4108/icst.collaboratecom.2010.33
Abstract
Peer reviewing is an important form of collaborative work that is used for quality assurance in science and in other domains like software development and knowledge management. Review ratings by authors have potential to improve the quality of peer reviews, by giving way to remuneration of good reviews. A significant problem, however, is that authors' perception is hardly neutral, but might be affected by the reviews. To gain insight into their perception of peer reviews, we have conducted a survey among the authors of papers submitted to a peer-reviewed computer science conference. One of our findings is that authors are satisfied with reviews whose comments they deem helpful, and when they feel that the reviewer has made an effort to understand the paper. Suprisingly, these results hold when controlled for the score given by the reviewer. Based on the study results, we discuss the suitability of author ratings to identify high-quality reviews. We describe a remuneration function for reviews based on author ratings that aims to neutralize the effects of review scores.