Finding an optimal balance between agreement and performance in an online reciprocal peer evaluation system

Kwangsu Cho, Christian D. Schunn

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

Consistent with movements integrating performance evaluation and improvement, the current study examines an assumption that more peer raters will produce better results or the so-called maxima strategy. This study examines the maxima strategy from both the agreement and performance perspectives with the intent of examining the role of feedback information in reliability and performance through an optimal number of peer raters per student ratee. It was found that the maxima strategy works consistently from agreement perspectives, whereas the relationship between the maxima strategy and student performance improvement follows an inverted U-shaped function. Accordingly, we recommend that the number of peer raters needs to be decided according to the optimal balance between reliability and performance, which maximizes ratees’ performances without sacrificing evaluation reliability.

Original languageEnglish
Pages (from-to)94-101
Number of pages8
JournalStudies in Educational Evaluation
Volume56
DOIs
Publication statusPublished - 2018 Mar 1

Fingerprint

evaluation
performance
student

All Science Journal Classification (ASJC) codes

  • Education

Cite this

@article{2c6d29fa333b4534825ca985712ba9cc,
title = "Finding an optimal balance between agreement and performance in an online reciprocal peer evaluation system",
abstract = "Consistent with movements integrating performance evaluation and improvement, the current study examines an assumption that more peer raters will produce better results or the so-called maxima strategy. This study examines the maxima strategy from both the agreement and performance perspectives with the intent of examining the role of feedback information in reliability and performance through an optimal number of peer raters per student ratee. It was found that the maxima strategy works consistently from agreement perspectives, whereas the relationship between the maxima strategy and student performance improvement follows an inverted U-shaped function. Accordingly, we recommend that the number of peer raters needs to be decided according to the optimal balance between reliability and performance, which maximizes ratees’ performances without sacrificing evaluation reliability.",
author = "Kwangsu Cho and Schunn, {Christian D.}",
year = "2018",
month = "3",
day = "1",
doi = "10.1016/j.stueduc.2017.12.001",
language = "English",
volume = "56",
pages = "94--101",
journal = "Studies in Educational Evaluation",
issn = "0191-491X",
publisher = "Elsevier Limited",

}

Finding an optimal balance between agreement and performance in an online reciprocal peer evaluation system. / Cho, Kwangsu; Schunn, Christian D.

In: Studies in Educational Evaluation, Vol. 56, 01.03.2018, p. 94-101.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Finding an optimal balance between agreement and performance in an online reciprocal peer evaluation system

AU - Cho, Kwangsu

AU - Schunn, Christian D.

PY - 2018/3/1

Y1 - 2018/3/1

N2 - Consistent with movements integrating performance evaluation and improvement, the current study examines an assumption that more peer raters will produce better results or the so-called maxima strategy. This study examines the maxima strategy from both the agreement and performance perspectives with the intent of examining the role of feedback information in reliability and performance through an optimal number of peer raters per student ratee. It was found that the maxima strategy works consistently from agreement perspectives, whereas the relationship between the maxima strategy and student performance improvement follows an inverted U-shaped function. Accordingly, we recommend that the number of peer raters needs to be decided according to the optimal balance between reliability and performance, which maximizes ratees’ performances without sacrificing evaluation reliability.

AB - Consistent with movements integrating performance evaluation and improvement, the current study examines an assumption that more peer raters will produce better results or the so-called maxima strategy. This study examines the maxima strategy from both the agreement and performance perspectives with the intent of examining the role of feedback information in reliability and performance through an optimal number of peer raters per student ratee. It was found that the maxima strategy works consistently from agreement perspectives, whereas the relationship between the maxima strategy and student performance improvement follows an inverted U-shaped function. Accordingly, we recommend that the number of peer raters needs to be decided according to the optimal balance between reliability and performance, which maximizes ratees’ performances without sacrificing evaluation reliability.

UR - http://www.scopus.com/inward/record.url?scp=85036616434&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85036616434&partnerID=8YFLogxK

U2 - 10.1016/j.stueduc.2017.12.001

DO - 10.1016/j.stueduc.2017.12.001

M3 - Article

AN - SCOPUS:85036616434

VL - 56

SP - 94

EP - 101

JO - Studies in Educational Evaluation

JF - Studies in Educational Evaluation

SN - 0191-491X

ER -