Deep Visual Saliency on Stereoscopic Images

Anh Duc Nguyen, Jongyoo Kim, Heeseok Oh, Haksub Kim, Weisi Lin, Sanghoon Lee

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

Visual saliency on stereoscopic 3D (S3D) images has been shown to be heavily influenced by image quality. Hence, this dependency is an important factor in image quality prediction, image restoration and discomfort reduction, but it is still very difficult to predict such a nonlinear relation in images. In addition, most algorithms specialized in detecting visual saliency on pristine images may unsurprisingly fail when facing distorted images. In this paper, we investigate a deep learning scheme named Deep Visual Saliency (DeepVS) to achieve a more accurate and reliable saliency predictor even in the presence of distortions. Since visual saliency is influenced by low-level features (contrast, luminance, and depth information) from a psychophysical point of view, we propose seven low-level features derived from S3D image pairs and utilize them in the context of deep learning to detect visual attention adaptively to human perception. During analysis, it turns out that the low-level features play a role to extract distortion and saliency information. To construct saliency predictors, we weight and model the human visual saliency through two different network architectures, a regression and a fully convolutional neural networks. Our results from thorough experiments confirm that the predicted saliency maps are up to 70% correlated with human gaze patterns, which emphasize the need for the hand-crafted features as input to deep neural networks in S3D saliency detection.

Original languageEnglish
Article number8520871
Pages (from-to)1939-1953
Number of pages15
JournalIEEE Transactions on Image Processing
Volume28
Issue number4
DOIs
Publication statusPublished - 2019 Apr

Fingerprint

Image quality
Image reconstruction
Network architecture
Luminance
Neural networks
Experiments
Deep learning
Deep neural networks

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Graphics and Computer-Aided Design

Cite this

Nguyen, A. D., Kim, J., Oh, H., Kim, H., Lin, W., & Lee, S. (2019). Deep Visual Saliency on Stereoscopic Images. IEEE Transactions on Image Processing, 28(4), 1939-1953. [8520871]. https://doi.org/10.1109/TIP.2018.2879408
Nguyen, Anh Duc ; Kim, Jongyoo ; Oh, Heeseok ; Kim, Haksub ; Lin, Weisi ; Lee, Sanghoon. / Deep Visual Saliency on Stereoscopic Images. In: IEEE Transactions on Image Processing. 2019 ; Vol. 28, No. 4. pp. 1939-1953.
@article{e5adb8e15632418bac86832390c954f6,
title = "Deep Visual Saliency on Stereoscopic Images",
abstract = "Visual saliency on stereoscopic 3D (S3D) images has been shown to be heavily influenced by image quality. Hence, this dependency is an important factor in image quality prediction, image restoration and discomfort reduction, but it is still very difficult to predict such a nonlinear relation in images. In addition, most algorithms specialized in detecting visual saliency on pristine images may unsurprisingly fail when facing distorted images. In this paper, we investigate a deep learning scheme named Deep Visual Saliency (DeepVS) to achieve a more accurate and reliable saliency predictor even in the presence of distortions. Since visual saliency is influenced by low-level features (contrast, luminance, and depth information) from a psychophysical point of view, we propose seven low-level features derived from S3D image pairs and utilize them in the context of deep learning to detect visual attention adaptively to human perception. During analysis, it turns out that the low-level features play a role to extract distortion and saliency information. To construct saliency predictors, we weight and model the human visual saliency through two different network architectures, a regression and a fully convolutional neural networks. Our results from thorough experiments confirm that the predicted saliency maps are up to 70{\%} correlated with human gaze patterns, which emphasize the need for the hand-crafted features as input to deep neural networks in S3D saliency detection.",
author = "Nguyen, {Anh Duc} and Jongyoo Kim and Heeseok Oh and Haksub Kim and Weisi Lin and Sanghoon Lee",
year = "2019",
month = "4",
doi = "10.1109/TIP.2018.2879408",
language = "English",
volume = "28",
pages = "1939--1953",
journal = "IEEE Transactions on Image Processing",
issn = "1057-7149",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "4",

}

Nguyen, AD, Kim, J, Oh, H, Kim, H, Lin, W & Lee, S 2019, 'Deep Visual Saliency on Stereoscopic Images', IEEE Transactions on Image Processing, vol. 28, no. 4, 8520871, pp. 1939-1953. https://doi.org/10.1109/TIP.2018.2879408

Deep Visual Saliency on Stereoscopic Images. / Nguyen, Anh Duc; Kim, Jongyoo; Oh, Heeseok; Kim, Haksub; Lin, Weisi; Lee, Sanghoon.

In: IEEE Transactions on Image Processing, Vol. 28, No. 4, 8520871, 04.2019, p. 1939-1953.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Deep Visual Saliency on Stereoscopic Images

AU - Nguyen, Anh Duc

AU - Kim, Jongyoo

AU - Oh, Heeseok

AU - Kim, Haksub

AU - Lin, Weisi

AU - Lee, Sanghoon

PY - 2019/4

Y1 - 2019/4

N2 - Visual saliency on stereoscopic 3D (S3D) images has been shown to be heavily influenced by image quality. Hence, this dependency is an important factor in image quality prediction, image restoration and discomfort reduction, but it is still very difficult to predict such a nonlinear relation in images. In addition, most algorithms specialized in detecting visual saliency on pristine images may unsurprisingly fail when facing distorted images. In this paper, we investigate a deep learning scheme named Deep Visual Saliency (DeepVS) to achieve a more accurate and reliable saliency predictor even in the presence of distortions. Since visual saliency is influenced by low-level features (contrast, luminance, and depth information) from a psychophysical point of view, we propose seven low-level features derived from S3D image pairs and utilize them in the context of deep learning to detect visual attention adaptively to human perception. During analysis, it turns out that the low-level features play a role to extract distortion and saliency information. To construct saliency predictors, we weight and model the human visual saliency through two different network architectures, a regression and a fully convolutional neural networks. Our results from thorough experiments confirm that the predicted saliency maps are up to 70% correlated with human gaze patterns, which emphasize the need for the hand-crafted features as input to deep neural networks in S3D saliency detection.

AB - Visual saliency on stereoscopic 3D (S3D) images has been shown to be heavily influenced by image quality. Hence, this dependency is an important factor in image quality prediction, image restoration and discomfort reduction, but it is still very difficult to predict such a nonlinear relation in images. In addition, most algorithms specialized in detecting visual saliency on pristine images may unsurprisingly fail when facing distorted images. In this paper, we investigate a deep learning scheme named Deep Visual Saliency (DeepVS) to achieve a more accurate and reliable saliency predictor even in the presence of distortions. Since visual saliency is influenced by low-level features (contrast, luminance, and depth information) from a psychophysical point of view, we propose seven low-level features derived from S3D image pairs and utilize them in the context of deep learning to detect visual attention adaptively to human perception. During analysis, it turns out that the low-level features play a role to extract distortion and saliency information. To construct saliency predictors, we weight and model the human visual saliency through two different network architectures, a regression and a fully convolutional neural networks. Our results from thorough experiments confirm that the predicted saliency maps are up to 70% correlated with human gaze patterns, which emphasize the need for the hand-crafted features as input to deep neural networks in S3D saliency detection.

UR - http://www.scopus.com/inward/record.url?scp=85056147202&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85056147202&partnerID=8YFLogxK

U2 - 10.1109/TIP.2018.2879408

DO - 10.1109/TIP.2018.2879408

M3 - Article

AN - SCOPUS:85056147202

VL - 28

SP - 1939

EP - 1953

JO - IEEE Transactions on Image Processing

JF - IEEE Transactions on Image Processing

SN - 1057-7149

IS - 4

M1 - 8520871

ER -