Paired comparison-based subjective quality assessment of stereoscopic images

Jong-Seok Lee, Lutz Goldmann, Touradj Ebrahimi

Research output: Contribution to journalArticle

28 Citations (Scopus)

Abstract

As 3D image and video content has gained significant popularity, subjective 3D quality assessment has become an important issue for the creation, processing, and distribution of high quality 3D content. Reliable subjective quality assessment of 3D content is often difficult due to the subjects' limited 3D experience, the interaction of multiple quality factors, minor quality differences between stimuli, etc. Among subjective evaluation methodologies, paired comparison has the advantage of improved simplicity and reliability, which can be useful to tackle the aforementioned difficulties. In this paper, we propose a new method to analyze the results of paired comparison-based subjective tests. We assume that ties convey information about the significance of quality score differences between two stimuli. Then, a maximum likelihood estimation is performed to obtain confidence intervals providing intuitive measures of significance of the quality differences. We describe the complete test procedure using the proposed method, from subjective experiment design to outlier detection and score analysis for 3D image quality assessment. Especially, we design the test procedure in a way that quality comparison across different contents is enabled while the number of pair-wise comparisons is minimized. Experimental results on a stereoscopic image database with varying camera distances demonstrate the usefulness of the proposed method and enhanced quality discriminability of paired comparison in comparison to the conventional single stimulus methodology.

Original languageEnglish
Pages (from-to)31-48
Number of pages18
JournalMultimedia Tools and Applications
Volume67
Issue number1
DOIs
Publication statusPublished - 2013 Nov 1

Fingerprint

Maximum likelihood estimation
Image quality
Cameras
Processing
Experiments

All Science Journal Classification (ASJC) codes

  • Software
  • Media Technology
  • Hardware and Architecture
  • Computer Networks and Communications

Cite this

Lee, Jong-Seok ; Goldmann, Lutz ; Ebrahimi, Touradj. / Paired comparison-based subjective quality assessment of stereoscopic images. In: Multimedia Tools and Applications. 2013 ; Vol. 67, No. 1. pp. 31-48.
@article{a7f0d5884b8f4cd2939b006cff59feb8,
title = "Paired comparison-based subjective quality assessment of stereoscopic images",
abstract = "As 3D image and video content has gained significant popularity, subjective 3D quality assessment has become an important issue for the creation, processing, and distribution of high quality 3D content. Reliable subjective quality assessment of 3D content is often difficult due to the subjects' limited 3D experience, the interaction of multiple quality factors, minor quality differences between stimuli, etc. Among subjective evaluation methodologies, paired comparison has the advantage of improved simplicity and reliability, which can be useful to tackle the aforementioned difficulties. In this paper, we propose a new method to analyze the results of paired comparison-based subjective tests. We assume that ties convey information about the significance of quality score differences between two stimuli. Then, a maximum likelihood estimation is performed to obtain confidence intervals providing intuitive measures of significance of the quality differences. We describe the complete test procedure using the proposed method, from subjective experiment design to outlier detection and score analysis for 3D image quality assessment. Especially, we design the test procedure in a way that quality comparison across different contents is enabled while the number of pair-wise comparisons is minimized. Experimental results on a stereoscopic image database with varying camera distances demonstrate the usefulness of the proposed method and enhanced quality discriminability of paired comparison in comparison to the conventional single stimulus methodology.",
author = "Jong-Seok Lee and Lutz Goldmann and Touradj Ebrahimi",
year = "2013",
month = "11",
day = "1",
doi = "10.1007/s11042-012-1011-6",
language = "English",
volume = "67",
pages = "31--48",
journal = "Multimedia Tools and Applications",
issn = "1380-7501",
publisher = "Springer Netherlands",
number = "1",

}

Paired comparison-based subjective quality assessment of stereoscopic images. / Lee, Jong-Seok; Goldmann, Lutz; Ebrahimi, Touradj.

In: Multimedia Tools and Applications, Vol. 67, No. 1, 01.11.2013, p. 31-48.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Paired comparison-based subjective quality assessment of stereoscopic images

AU - Lee, Jong-Seok

AU - Goldmann, Lutz

AU - Ebrahimi, Touradj

PY - 2013/11/1

Y1 - 2013/11/1

N2 - As 3D image and video content has gained significant popularity, subjective 3D quality assessment has become an important issue for the creation, processing, and distribution of high quality 3D content. Reliable subjective quality assessment of 3D content is often difficult due to the subjects' limited 3D experience, the interaction of multiple quality factors, minor quality differences between stimuli, etc. Among subjective evaluation methodologies, paired comparison has the advantage of improved simplicity and reliability, which can be useful to tackle the aforementioned difficulties. In this paper, we propose a new method to analyze the results of paired comparison-based subjective tests. We assume that ties convey information about the significance of quality score differences between two stimuli. Then, a maximum likelihood estimation is performed to obtain confidence intervals providing intuitive measures of significance of the quality differences. We describe the complete test procedure using the proposed method, from subjective experiment design to outlier detection and score analysis for 3D image quality assessment. Especially, we design the test procedure in a way that quality comparison across different contents is enabled while the number of pair-wise comparisons is minimized. Experimental results on a stereoscopic image database with varying camera distances demonstrate the usefulness of the proposed method and enhanced quality discriminability of paired comparison in comparison to the conventional single stimulus methodology.

AB - As 3D image and video content has gained significant popularity, subjective 3D quality assessment has become an important issue for the creation, processing, and distribution of high quality 3D content. Reliable subjective quality assessment of 3D content is often difficult due to the subjects' limited 3D experience, the interaction of multiple quality factors, minor quality differences between stimuli, etc. Among subjective evaluation methodologies, paired comparison has the advantage of improved simplicity and reliability, which can be useful to tackle the aforementioned difficulties. In this paper, we propose a new method to analyze the results of paired comparison-based subjective tests. We assume that ties convey information about the significance of quality score differences between two stimuli. Then, a maximum likelihood estimation is performed to obtain confidence intervals providing intuitive measures of significance of the quality differences. We describe the complete test procedure using the proposed method, from subjective experiment design to outlier detection and score analysis for 3D image quality assessment. Especially, we design the test procedure in a way that quality comparison across different contents is enabled while the number of pair-wise comparisons is minimized. Experimental results on a stereoscopic image database with varying camera distances demonstrate the usefulness of the proposed method and enhanced quality discriminability of paired comparison in comparison to the conventional single stimulus methodology.

UR - http://www.scopus.com/inward/record.url?scp=84881150185&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84881150185&partnerID=8YFLogxK

U2 - 10.1007/s11042-012-1011-6

DO - 10.1007/s11042-012-1011-6

M3 - Article

VL - 67

SP - 31

EP - 48

JO - Multimedia Tools and Applications

JF - Multimedia Tools and Applications

SN - 1380-7501

IS - 1

ER -