6th International ICST Conference on Collaborative Computing: Networking, Applications, Worksharing

Research Article

Reviewing the reviewers: A study of author perception on peer reviews in computer science

Download635 downloads
  • @INPROCEEDINGS{10.4108/icst.collaboratecom.2010.33,
        author={Conny K\'{y}hne and Klemens B\o{}hm and Jing Zhi Yue},
        title={Reviewing the reviewers: A study of author perception on peer reviews in computer science},
        proceedings={6th International ICST Conference on Collaborative Computing: Networking, Applications, Worksharing},
        publisher={IEEE},
        proceedings_a={COLLABORATECOM},
        year={2011},
        month={5},
        keywords={Computer aided software engineering},
        doi={10.4108/icst.collaboratecom.2010.33}
    }
    
  • Conny Kühne
    Klemens Böhm
    Jing Zhi Yue
    Year: 2011
    Reviewing the reviewers: A study of author perception on peer reviews in computer science
    COLLABORATECOM
    ICST
    DOI: 10.4108/icst.collaboratecom.2010.33
Conny Kühne1,*, Klemens Böhm1,*, Jing Zhi Yue1,*
  • 1: Karlsruhe Institute of Technology (KIT), Germany
*Contact email: conny.kuehne@kit.edu, klemens.boehm@kit.edu, jing.yue@kit.edu

Abstract

Peer reviewing is an important form of collaborative work that is used for quality assurance in science and in other domains like software development and knowledge management. Review ratings by authors have potential to improve the quality of peer reviews, by giving way to remuneration of good reviews. A significant problem, however, is that authors' perception is hardly neutral, but might be affected by the reviews. To gain insight into their perception of peer reviews, we have conducted a survey among the authors of papers submitted to a peer-reviewed computer science conference. One of our findings is that authors are satisfied with reviews whose comments they deem helpful, and when they feel that the reviewer has made an effort to understand the paper. Suprisingly, these results hold when controlled for the score given by the reviewer. Based on the study results, we discuss the suitability of author ratings to identify high-quality reviews. We describe a remuneration function for reviews based on author ratings that aims to neutralize the effects of review scores.