On designing paired comparison experiments for subjective multimedia quality assessment

Research output: Contribution to journalArticle

17 Citations (Scopus)

Abstract

This paper investigates the issue of designing paired comparison-based subjective quality assessment experiments for reliable results. In particular, the convergence behavior of the quality scores estimated from paired comparison results is considered. Via an extensive computer simulation experiment, the estimation performance in terms of the root mean squared error, the rank order correlation coefficient, and the change of the estimated scores with respect to the number of subjects are mathematically modeled. Furthermore, it is confirmed that the models coincide with the theoretical convergence behavior. Issues such as the effect of human errors and the underlying distribution of the true quality scores are also examined.

Original languageEnglish
Article number6675876
Pages (from-to)564-571
Number of pages8
JournalIEEE Transactions on Multimedia
Volume16
Issue number2
DOIs
Publication statusPublished - 2014 Feb 1

Fingerprint

Experiments
Computer simulation

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Media Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering

Cite this

@article{3041f0cb4ac0475fa132717eea026728,
title = "On designing paired comparison experiments for subjective multimedia quality assessment",
abstract = "This paper investigates the issue of designing paired comparison-based subjective quality assessment experiments for reliable results. In particular, the convergence behavior of the quality scores estimated from paired comparison results is considered. Via an extensive computer simulation experiment, the estimation performance in terms of the root mean squared error, the rank order correlation coefficient, and the change of the estimated scores with respect to the number of subjects are mathematically modeled. Furthermore, it is confirmed that the models coincide with the theoretical convergence behavior. Issues such as the effect of human errors and the underlying distribution of the true quality scores are also examined.",
author = "Jong-Seok Lee",
year = "2014",
month = "2",
day = "1",
doi = "10.1109/TMM.2013.2292590",
language = "English",
volume = "16",
pages = "564--571",
journal = "IEEE Transactions on Multimedia",
issn = "1520-9210",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "2",

}

On designing paired comparison experiments for subjective multimedia quality assessment. / Lee, Jong-Seok.

In: IEEE Transactions on Multimedia, Vol. 16, No. 2, 6675876, 01.02.2014, p. 564-571.

Research output: Contribution to journalArticle

TY - JOUR

T1 - On designing paired comparison experiments for subjective multimedia quality assessment

AU - Lee, Jong-Seok

PY - 2014/2/1

Y1 - 2014/2/1

N2 - This paper investigates the issue of designing paired comparison-based subjective quality assessment experiments for reliable results. In particular, the convergence behavior of the quality scores estimated from paired comparison results is considered. Via an extensive computer simulation experiment, the estimation performance in terms of the root mean squared error, the rank order correlation coefficient, and the change of the estimated scores with respect to the number of subjects are mathematically modeled. Furthermore, it is confirmed that the models coincide with the theoretical convergence behavior. Issues such as the effect of human errors and the underlying distribution of the true quality scores are also examined.

AB - This paper investigates the issue of designing paired comparison-based subjective quality assessment experiments for reliable results. In particular, the convergence behavior of the quality scores estimated from paired comparison results is considered. Via an extensive computer simulation experiment, the estimation performance in terms of the root mean squared error, the rank order correlation coefficient, and the change of the estimated scores with respect to the number of subjects are mathematically modeled. Furthermore, it is confirmed that the models coincide with the theoretical convergence behavior. Issues such as the effect of human errors and the underlying distribution of the true quality scores are also examined.

UR - http://www.scopus.com/inward/record.url?scp=84893421305&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84893421305&partnerID=8YFLogxK

U2 - 10.1109/TMM.2013.2292590

DO - 10.1109/TMM.2013.2292590

M3 - Article

VL - 16

SP - 564

EP - 571

JO - IEEE Transactions on Multimedia

JF - IEEE Transactions on Multimedia

SN - 1520-9210

IS - 2

M1 - 6675876

ER -