Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation

Heeseok Oh, Sewoong Ahn, Jongyoo Kim, Sanghoon Lee

Research output: Contribution to journalArticle

19 Citations (Scopus)

Abstract

Previously, no-reference (NR) stereoscopic 3D (S3D) image quality assessment (IQA) algorithms have been limited to the extraction of reliable hand-crafted features based on an understanding of the insufficiently revealed human visual system or natural scene statistics. Furthermore, compared with full-reference (FR) S3D IQA metrics, it is difficult to achieve competitive quality score predictions using the extracted features, which are not optimized with respect to human opinion. To cope with this limitation of the conventional approach, we introduce a novel deep learning scheme for NR S3D IQA in terms of local to global feature aggregation. A deep convolutional neural network (CNN) model is trained in a supervised manner through two-step regression. First, to overcome the lack of training data, local patch-based CNNs are modeled, and the FR S3D IQA metric is used to approximate a reference ground-truth for training the CNNs. The automatically extracted local abstractions are aggregated into global features by inserting an aggregation layer in the deep structure. The locally trained model parameters are then updated iteratively using supervised global labeling, i.e., subjective mean opinion score (MOS). In particular, the proposed deep NR S3D image quality evaluator does not estimate the depth from a pair of S3D images. The S3D image quality scores predicted by the proposed method represent a significant improvement over those of previous NR S3D IQA algorithms. Indeed, the accuracy of the proposed method is competitive with FR S3D IQA metrics, having 91% correlation in terms of MOS.

Original languageEnglish
Article number7973187
Pages (from-to)4923-4936
Number of pages14
JournalIEEE Transactions on Image Processing
Volume26
Issue number10
DOIs
Publication statusPublished - 2017 Oct

Fingerprint

Image quality
Agglomeration
Labeling
Statistics
Neural networks

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Graphics and Computer-Aided Design

Cite this

Oh, Heeseok ; Ahn, Sewoong ; Kim, Jongyoo ; Lee, Sanghoon. / Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation. In: IEEE Transactions on Image Processing. 2017 ; Vol. 26, No. 10. pp. 4923-4936.
@article{fe4530d14b6f4694ba6867e8e1fe24c3,
title = "Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation",
abstract = "Previously, no-reference (NR) stereoscopic 3D (S3D) image quality assessment (IQA) algorithms have been limited to the extraction of reliable hand-crafted features based on an understanding of the insufficiently revealed human visual system or natural scene statistics. Furthermore, compared with full-reference (FR) S3D IQA metrics, it is difficult to achieve competitive quality score predictions using the extracted features, which are not optimized with respect to human opinion. To cope with this limitation of the conventional approach, we introduce a novel deep learning scheme for NR S3D IQA in terms of local to global feature aggregation. A deep convolutional neural network (CNN) model is trained in a supervised manner through two-step regression. First, to overcome the lack of training data, local patch-based CNNs are modeled, and the FR S3D IQA metric is used to approximate a reference ground-truth for training the CNNs. The automatically extracted local abstractions are aggregated into global features by inserting an aggregation layer in the deep structure. The locally trained model parameters are then updated iteratively using supervised global labeling, i.e., subjective mean opinion score (MOS). In particular, the proposed deep NR S3D image quality evaluator does not estimate the depth from a pair of S3D images. The S3D image quality scores predicted by the proposed method represent a significant improvement over those of previous NR S3D IQA algorithms. Indeed, the accuracy of the proposed method is competitive with FR S3D IQA metrics, having 91{\%} correlation in terms of MOS.",
author = "Heeseok Oh and Sewoong Ahn and Jongyoo Kim and Sanghoon Lee",
year = "2017",
month = "10",
doi = "10.1109/TIP.2017.2725584",
language = "English",
volume = "26",
pages = "4923--4936",
journal = "IEEE Transactions on Image Processing",
issn = "1057-7149",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "10",

}

Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation. / Oh, Heeseok; Ahn, Sewoong; Kim, Jongyoo; Lee, Sanghoon.

In: IEEE Transactions on Image Processing, Vol. 26, No. 10, 7973187, 10.2017, p. 4923-4936.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation

AU - Oh, Heeseok

AU - Ahn, Sewoong

AU - Kim, Jongyoo

AU - Lee, Sanghoon

PY - 2017/10

Y1 - 2017/10

N2 - Previously, no-reference (NR) stereoscopic 3D (S3D) image quality assessment (IQA) algorithms have been limited to the extraction of reliable hand-crafted features based on an understanding of the insufficiently revealed human visual system or natural scene statistics. Furthermore, compared with full-reference (FR) S3D IQA metrics, it is difficult to achieve competitive quality score predictions using the extracted features, which are not optimized with respect to human opinion. To cope with this limitation of the conventional approach, we introduce a novel deep learning scheme for NR S3D IQA in terms of local to global feature aggregation. A deep convolutional neural network (CNN) model is trained in a supervised manner through two-step regression. First, to overcome the lack of training data, local patch-based CNNs are modeled, and the FR S3D IQA metric is used to approximate a reference ground-truth for training the CNNs. The automatically extracted local abstractions are aggregated into global features by inserting an aggregation layer in the deep structure. The locally trained model parameters are then updated iteratively using supervised global labeling, i.e., subjective mean opinion score (MOS). In particular, the proposed deep NR S3D image quality evaluator does not estimate the depth from a pair of S3D images. The S3D image quality scores predicted by the proposed method represent a significant improvement over those of previous NR S3D IQA algorithms. Indeed, the accuracy of the proposed method is competitive with FR S3D IQA metrics, having 91% correlation in terms of MOS.

AB - Previously, no-reference (NR) stereoscopic 3D (S3D) image quality assessment (IQA) algorithms have been limited to the extraction of reliable hand-crafted features based on an understanding of the insufficiently revealed human visual system or natural scene statistics. Furthermore, compared with full-reference (FR) S3D IQA metrics, it is difficult to achieve competitive quality score predictions using the extracted features, which are not optimized with respect to human opinion. To cope with this limitation of the conventional approach, we introduce a novel deep learning scheme for NR S3D IQA in terms of local to global feature aggregation. A deep convolutional neural network (CNN) model is trained in a supervised manner through two-step regression. First, to overcome the lack of training data, local patch-based CNNs are modeled, and the FR S3D IQA metric is used to approximate a reference ground-truth for training the CNNs. The automatically extracted local abstractions are aggregated into global features by inserting an aggregation layer in the deep structure. The locally trained model parameters are then updated iteratively using supervised global labeling, i.e., subjective mean opinion score (MOS). In particular, the proposed deep NR S3D image quality evaluator does not estimate the depth from a pair of S3D images. The S3D image quality scores predicted by the proposed method represent a significant improvement over those of previous NR S3D IQA algorithms. Indeed, the accuracy of the proposed method is competitive with FR S3D IQA metrics, having 91% correlation in terms of MOS.

UR - http://www.scopus.com/inward/record.url?scp=85023630714&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85023630714&partnerID=8YFLogxK

U2 - 10.1109/TIP.2017.2725584

DO - 10.1109/TIP.2017.2725584

M3 - Article

C2 - 28708557

AN - SCOPUS:85023630714

VL - 26

SP - 4923

EP - 4936

JO - IEEE Transactions on Image Processing

JF - IEEE Transactions on Image Processing

SN - 1057-7149

IS - 10

M1 - 7973187

ER -