Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives

Kwangsu Cho, Christian D. Schunn, Roy W. Wilson

Research output: Contribution to journalReview article

170 Citations (Scopus)

Abstract

Although peer reviewing of writing is a way to create more writing opportunities in college and university settings, the validity and reliability of peer-generated grades are a major concern. This study investigated the validity and reliability of peer-generated writing grades of 708 students across 16 different courses from 4 universities in a particular scaffolded reviewing context: Students were given guidance on peer assessment, used carefully constructed rubrics, and were provided clear incentives to take the assessment task seriously. Distinguishing between instructor and student perspectives of reliability and validity, the analyses suggest that the aggregate ratings of at least 4 peers on a piece of writing are both highly reliable and as valid as instructor ratings while (paradoxically) producing very low estimates of reliability and validity from the student perspective. The results suggest that instructor concerns about peer evaluation reliability and validity should not be a barrier to implementing peer evaluations, at least with appropriate scaffolds. Future research needs to investigate how to address student concerns about reliability and validity and to identify scaffolds that may ensure high levels of reliability and validity. (PsycINFO Database Record (c) 2006 APA, all rights reserved).

Original languageEnglish
Pages (from-to)891-901
Number of pages11
JournalJournal of Educational Psychology
Volume98
Issue number4
DOIs
Publication statusPublished - 2006 Nov 1

Fingerprint

Reproducibility of Results
instructor
Students
student
rating
university
evaluation
Motivation
incentive

All Science Journal Classification (ASJC) codes

  • Education
  • Developmental and Educational Psychology

Cite this

Cho, Kwangsu ; Schunn, Christian D. ; Wilson, Roy W. / Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives. In: Journal of Educational Psychology. 2006 ; Vol. 98, No. 4. pp. 891-901.
@article{9e8082057453463484abe5f93ab0865d,
title = "Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives",
abstract = "Although peer reviewing of writing is a way to create more writing opportunities in college and university settings, the validity and reliability of peer-generated grades are a major concern. This study investigated the validity and reliability of peer-generated writing grades of 708 students across 16 different courses from 4 universities in a particular scaffolded reviewing context: Students were given guidance on peer assessment, used carefully constructed rubrics, and were provided clear incentives to take the assessment task seriously. Distinguishing between instructor and student perspectives of reliability and validity, the analyses suggest that the aggregate ratings of at least 4 peers on a piece of writing are both highly reliable and as valid as instructor ratings while (paradoxically) producing very low estimates of reliability and validity from the student perspective. The results suggest that instructor concerns about peer evaluation reliability and validity should not be a barrier to implementing peer evaluations, at least with appropriate scaffolds. Future research needs to investigate how to address student concerns about reliability and validity and to identify scaffolds that may ensure high levels of reliability and validity. (PsycINFO Database Record (c) 2006 APA, all rights reserved).",
author = "Kwangsu Cho and Schunn, {Christian D.} and Wilson, {Roy W.}",
year = "2006",
month = "11",
day = "1",
doi = "10.1037/0022-0663.98.4.891",
language = "English",
volume = "98",
pages = "891--901",
journal = "Journal of Educational Psychology",
issn = "0022-0663",
publisher = "American Psychological Association Inc.",
number = "4",

}

Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives. / Cho, Kwangsu; Schunn, Christian D.; Wilson, Roy W.

In: Journal of Educational Psychology, Vol. 98, No. 4, 01.11.2006, p. 891-901.

Research output: Contribution to journalReview article

TY - JOUR

T1 - Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives

AU - Cho, Kwangsu

AU - Schunn, Christian D.

AU - Wilson, Roy W.

PY - 2006/11/1

Y1 - 2006/11/1

N2 - Although peer reviewing of writing is a way to create more writing opportunities in college and university settings, the validity and reliability of peer-generated grades are a major concern. This study investigated the validity and reliability of peer-generated writing grades of 708 students across 16 different courses from 4 universities in a particular scaffolded reviewing context: Students were given guidance on peer assessment, used carefully constructed rubrics, and were provided clear incentives to take the assessment task seriously. Distinguishing between instructor and student perspectives of reliability and validity, the analyses suggest that the aggregate ratings of at least 4 peers on a piece of writing are both highly reliable and as valid as instructor ratings while (paradoxically) producing very low estimates of reliability and validity from the student perspective. The results suggest that instructor concerns about peer evaluation reliability and validity should not be a barrier to implementing peer evaluations, at least with appropriate scaffolds. Future research needs to investigate how to address student concerns about reliability and validity and to identify scaffolds that may ensure high levels of reliability and validity. (PsycINFO Database Record (c) 2006 APA, all rights reserved).

AB - Although peer reviewing of writing is a way to create more writing opportunities in college and university settings, the validity and reliability of peer-generated grades are a major concern. This study investigated the validity and reliability of peer-generated writing grades of 708 students across 16 different courses from 4 universities in a particular scaffolded reviewing context: Students were given guidance on peer assessment, used carefully constructed rubrics, and were provided clear incentives to take the assessment task seriously. Distinguishing between instructor and student perspectives of reliability and validity, the analyses suggest that the aggregate ratings of at least 4 peers on a piece of writing are both highly reliable and as valid as instructor ratings while (paradoxically) producing very low estimates of reliability and validity from the student perspective. The results suggest that instructor concerns about peer evaluation reliability and validity should not be a barrier to implementing peer evaluations, at least with appropriate scaffolds. Future research needs to investigate how to address student concerns about reliability and validity and to identify scaffolds that may ensure high levels of reliability and validity. (PsycINFO Database Record (c) 2006 APA, all rights reserved).

UR - http://www.scopus.com/inward/record.url?scp=33751015138&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33751015138&partnerID=8YFLogxK

U2 - 10.1037/0022-0663.98.4.891

DO - 10.1037/0022-0663.98.4.891

M3 - Review article

VL - 98

SP - 891

EP - 901

JO - Journal of Educational Psychology

JF - Journal of Educational Psychology

SN - 0022-0663

IS - 4

ER -