A new analysis method for paired comparison and its application to 3D quality assessment

Jong-Seok Lee, Lutz Goldmann, Touradj Ebrahimi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

9 Citations (Scopus)

Abstract

Among various subjective quality evaluation methodologies, paired comparison has the advantage of improved simplicity of the subjects'evaluation task due to simplified rating scales and direct comparison of two stimuli. Thus, it may lead to more reliable results when individual quality levels are difficult to define, quality differences between stimuli are small or multiple quality factors are involved. This paper proposes a new method to analyze results of paired comparison-based subjective tests. By assuming that ties convey information about significant differences between two stimuli being compared, the confidence intervals for the quality scores are estimated using a maximum likelihood criterion, which enables us to intuitively examine the significance of quality score differences. We describe the complete test methodology including the test procedure, outlier detection and score analysis applied to quality assessment of 3D images acquired using varying camera distances. Experimental results demonstrate the usefulness of the proposed analysis method, as well as the enhanced quality discriminability of the paired comparison methodology in comparison to the conventional single stimulus methodology.

Original languageEnglish
Title of host publicationMM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops
Pages1281-1284
Number of pages4
DOIs
Publication statusPublished - 2011 Dec 29
Event19th ACM International Conference on Multimedia ACM Multimedia 2011, MM'11 - Scottsdale, AZ, United States
Duration: 2011 Nov 282011 Dec 1

Publication series

NameMM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops

Other

Other19th ACM International Conference on Multimedia ACM Multimedia 2011, MM'11
CountryUnited States
CityScottsdale, AZ
Period11/11/2811/12/1

Fingerprint

Maximum likelihood
Cameras

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction

Cite this

Lee, J-S., Goldmann, L., & Ebrahimi, T. (2011). A new analysis method for paired comparison and its application to 3D quality assessment. In MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops (pp. 1281-1284). (MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops). https://doi.org/10.1145/2072298.2071994
Lee, Jong-Seok ; Goldmann, Lutz ; Ebrahimi, Touradj. / A new analysis method for paired comparison and its application to 3D quality assessment. MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops. 2011. pp. 1281-1284 (MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops).
@inproceedings{3a418c914d7b4ef0a4cb7ff7a1240baa,
title = "A new analysis method for paired comparison and its application to 3D quality assessment",
abstract = "Among various subjective quality evaluation methodologies, paired comparison has the advantage of improved simplicity of the subjects'evaluation task due to simplified rating scales and direct comparison of two stimuli. Thus, it may lead to more reliable results when individual quality levels are difficult to define, quality differences between stimuli are small or multiple quality factors are involved. This paper proposes a new method to analyze results of paired comparison-based subjective tests. By assuming that ties convey information about significant differences between two stimuli being compared, the confidence intervals for the quality scores are estimated using a maximum likelihood criterion, which enables us to intuitively examine the significance of quality score differences. We describe the complete test methodology including the test procedure, outlier detection and score analysis applied to quality assessment of 3D images acquired using varying camera distances. Experimental results demonstrate the usefulness of the proposed analysis method, as well as the enhanced quality discriminability of the paired comparison methodology in comparison to the conventional single stimulus methodology.",
author = "Jong-Seok Lee and Lutz Goldmann and Touradj Ebrahimi",
year = "2011",
month = "12",
day = "29",
doi = "10.1145/2072298.2071994",
language = "English",
isbn = "9781450306164",
series = "MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops",
pages = "1281--1284",
booktitle = "MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops",

}

Lee, J-S, Goldmann, L & Ebrahimi, T 2011, A new analysis method for paired comparison and its application to 3D quality assessment. in MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops. MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops, pp. 1281-1284, 19th ACM International Conference on Multimedia ACM Multimedia 2011, MM'11, Scottsdale, AZ, United States, 11/11/28. https://doi.org/10.1145/2072298.2071994

A new analysis method for paired comparison and its application to 3D quality assessment. / Lee, Jong-Seok; Goldmann, Lutz; Ebrahimi, Touradj.

MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops. 2011. p. 1281-1284 (MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - A new analysis method for paired comparison and its application to 3D quality assessment

AU - Lee, Jong-Seok

AU - Goldmann, Lutz

AU - Ebrahimi, Touradj

PY - 2011/12/29

Y1 - 2011/12/29

N2 - Among various subjective quality evaluation methodologies, paired comparison has the advantage of improved simplicity of the subjects'evaluation task due to simplified rating scales and direct comparison of two stimuli. Thus, it may lead to more reliable results when individual quality levels are difficult to define, quality differences between stimuli are small or multiple quality factors are involved. This paper proposes a new method to analyze results of paired comparison-based subjective tests. By assuming that ties convey information about significant differences between two stimuli being compared, the confidence intervals for the quality scores are estimated using a maximum likelihood criterion, which enables us to intuitively examine the significance of quality score differences. We describe the complete test methodology including the test procedure, outlier detection and score analysis applied to quality assessment of 3D images acquired using varying camera distances. Experimental results demonstrate the usefulness of the proposed analysis method, as well as the enhanced quality discriminability of the paired comparison methodology in comparison to the conventional single stimulus methodology.

AB - Among various subjective quality evaluation methodologies, paired comparison has the advantage of improved simplicity of the subjects'evaluation task due to simplified rating scales and direct comparison of two stimuli. Thus, it may lead to more reliable results when individual quality levels are difficult to define, quality differences between stimuli are small or multiple quality factors are involved. This paper proposes a new method to analyze results of paired comparison-based subjective tests. By assuming that ties convey information about significant differences between two stimuli being compared, the confidence intervals for the quality scores are estimated using a maximum likelihood criterion, which enables us to intuitively examine the significance of quality score differences. We describe the complete test methodology including the test procedure, outlier detection and score analysis applied to quality assessment of 3D images acquired using varying camera distances. Experimental results demonstrate the usefulness of the proposed analysis method, as well as the enhanced quality discriminability of the paired comparison methodology in comparison to the conventional single stimulus methodology.

UR - http://www.scopus.com/inward/record.url?scp=84455205247&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84455205247&partnerID=8YFLogxK

U2 - 10.1145/2072298.2071994

DO - 10.1145/2072298.2071994

M3 - Conference contribution

SN - 9781450306164

T3 - MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops

SP - 1281

EP - 1284

BT - MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops

ER -

Lee J-S, Goldmann L, Ebrahimi T. A new analysis method for paired comparison and its application to 3D quality assessment. In MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops. 2011. p. 1281-1284. (MM'11 - Proceedings of the 2011 ACM Multimedia Conference and Co-Located Workshops). https://doi.org/10.1145/2072298.2071994