Motion Sickness Prediction in Stereoscopic Videos using 3D Convolutional Neural Networks

Tae Min Lee, Jong Chul Yoon, In Kwon Lee

Research output: Contribution to journalArticle

Abstract

In this paper, we propose a three-dimensional (3D) convolutional neural network (CNN)-based method for predicting the degree of motion sickness induced by a 360° stereoscopic video. We consider the user's eye movement as a new feature, in addition to the motion velocity and depth features of a video used in previous work. For this purpose, we use saliency, optical flow, and disparity maps of an input video, which represent eye movement, velocity, and depth, respectively, as the input of the 3D CNN. To train our machine-learning model, we extend the dataset established in the previous work using two data augmentation techniques: frame shifting and pixel shifting. Consequently, our model can predict the degree of motion sickness more precisely than the previous method, and the results have a more similar correlation to the distribution of ground-truth sickness.

Original languageEnglish
Article number8642906
Pages (from-to)1919-1927
Number of pages9
JournalIEEE Transactions on Visualization and Computer Graphics
Volume25
Issue number5
DOIs
Publication statusPublished - 2019 May 1

Fingerprint

Eye movements
Neural networks
Optical flows
Learning systems
Pixels

All Science Journal Classification (ASJC) codes

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design

Cite this

@article{6764ab12d7844f379aad4d0466f9cf79,
title = "Motion Sickness Prediction in Stereoscopic Videos using 3D Convolutional Neural Networks",
abstract = "In this paper, we propose a three-dimensional (3D) convolutional neural network (CNN)-based method for predicting the degree of motion sickness induced by a 360° stereoscopic video. We consider the user's eye movement as a new feature, in addition to the motion velocity and depth features of a video used in previous work. For this purpose, we use saliency, optical flow, and disparity maps of an input video, which represent eye movement, velocity, and depth, respectively, as the input of the 3D CNN. To train our machine-learning model, we extend the dataset established in the previous work using two data augmentation techniques: frame shifting and pixel shifting. Consequently, our model can predict the degree of motion sickness more precisely than the previous method, and the results have a more similar correlation to the distribution of ground-truth sickness.",
author = "Lee, {Tae Min} and Yoon, {Jong Chul} and Lee, {In Kwon}",
year = "2019",
month = "5",
day = "1",
doi = "10.1109/TVCG.2019.2899186",
language = "English",
volume = "25",
pages = "1919--1927",
journal = "IEEE Transactions on Visualization and Computer Graphics",
issn = "1077-2626",
publisher = "IEEE Computer Society",
number = "5",

}

Motion Sickness Prediction in Stereoscopic Videos using 3D Convolutional Neural Networks. / Lee, Tae Min; Yoon, Jong Chul; Lee, In Kwon.

In: IEEE Transactions on Visualization and Computer Graphics, Vol. 25, No. 5, 8642906, 01.05.2019, p. 1919-1927.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Motion Sickness Prediction in Stereoscopic Videos using 3D Convolutional Neural Networks

AU - Lee, Tae Min

AU - Yoon, Jong Chul

AU - Lee, In Kwon

PY - 2019/5/1

Y1 - 2019/5/1

N2 - In this paper, we propose a three-dimensional (3D) convolutional neural network (CNN)-based method for predicting the degree of motion sickness induced by a 360° stereoscopic video. We consider the user's eye movement as a new feature, in addition to the motion velocity and depth features of a video used in previous work. For this purpose, we use saliency, optical flow, and disparity maps of an input video, which represent eye movement, velocity, and depth, respectively, as the input of the 3D CNN. To train our machine-learning model, we extend the dataset established in the previous work using two data augmentation techniques: frame shifting and pixel shifting. Consequently, our model can predict the degree of motion sickness more precisely than the previous method, and the results have a more similar correlation to the distribution of ground-truth sickness.

AB - In this paper, we propose a three-dimensional (3D) convolutional neural network (CNN)-based method for predicting the degree of motion sickness induced by a 360° stereoscopic video. We consider the user's eye movement as a new feature, in addition to the motion velocity and depth features of a video used in previous work. For this purpose, we use saliency, optical flow, and disparity maps of an input video, which represent eye movement, velocity, and depth, respectively, as the input of the 3D CNN. To train our machine-learning model, we extend the dataset established in the previous work using two data augmentation techniques: frame shifting and pixel shifting. Consequently, our model can predict the degree of motion sickness more precisely than the previous method, and the results have a more similar correlation to the distribution of ground-truth sickness.

UR - http://www.scopus.com/inward/record.url?scp=85063798660&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063798660&partnerID=8YFLogxK

U2 - 10.1109/TVCG.2019.2899186

DO - 10.1109/TVCG.2019.2899186

M3 - Article

VL - 25

SP - 1919

EP - 1927

JO - IEEE Transactions on Visualization and Computer Graphics

JF - IEEE Transactions on Visualization and Computer Graphics

SN - 1077-2626

IS - 5

M1 - 8642906

ER -