Depth camera-based 3D hand gesture controls with immersive tactile feedback for natural mid-air gesture interactions

Kwangtaek Kim, Joongrock Kim, Jaesung Choi, Junghyun Kim, Sangyoun Lee

Research output: Contribution to journalArticle

22 Citations (Scopus)

Abstract

Vision-based hand gesture interactions are natural and intuitive when interacting with computers, since we naturally exploit gestures to communicate with other people. However, it is agreed that users suffer from discomfort and fatigue when using gesture-controlled interfaces, due to the lack of physical feedback. To solve the problem, we propose a novel complete solution of a hand gesture control system employing immersive tactile feedback to the user’s hand. For this goal, we first developed a fast and accurate hand-tracking algorithm with a Kinect sensor using the proposed MLBP (modified local binary pattern) that can efficiently analyze 3D shapes in depth images. The superiority of our tracking method was verified in terms of tracking accuracy and speed by comparing with existing methods, Natural Interaction Technology for End-user (NITE), 3D Hand Tracker and CamShift. As the second step, a new tactile feedback technology with a piezoelectric actuator has been developed and integrated into the developed hand tracking algorithm, including the DTW (dynamic time warping) gesture recognition algorithm for a complete solution of an immersive gesture control system. The quantitative and qualitative evaluations of the integrated system were conducted with human subjects, and the results demonstrate that our gesture control with tactile feedback is a promising technology compared to a vision-based gesture control system that has typically no feedback for the user’s gesture inputs. Our study provides researchers and designers with informative guidelines to develop more natural gesture control systems or immersive user interfaces with haptic feedback.

Original languageEnglish
Pages (from-to)1022-1046
Number of pages25
JournalSensors (Switzerland)
Volume15
Issue number1
DOIs
Publication statusPublished - 2015 Jan 8

Fingerprint

Gestures
Touch
Hand
Cameras
cameras
Air
Feedback
air
Control systems
interactions
Gesture recognition
Technology
piezoelectric actuators
Piezoelectric actuators
User interfaces
Fatigue of materials
evaluation
sensors
Sensors
Fatigue

All Science Journal Classification (ASJC) codes

  • Analytical Chemistry
  • Biochemistry
  • Atomic and Molecular Physics, and Optics
  • Instrumentation
  • Electrical and Electronic Engineering

Cite this

Kim, Kwangtaek ; Kim, Joongrock ; Choi, Jaesung ; Kim, Junghyun ; Lee, Sangyoun. / Depth camera-based 3D hand gesture controls with immersive tactile feedback for natural mid-air gesture interactions. In: Sensors (Switzerland). 2015 ; Vol. 15, No. 1. pp. 1022-1046.
@article{9fda16d0ad704998badc2903d6142653,
title = "Depth camera-based 3D hand gesture controls with immersive tactile feedback for natural mid-air gesture interactions",
abstract = "Vision-based hand gesture interactions are natural and intuitive when interacting with computers, since we naturally exploit gestures to communicate with other people. However, it is agreed that users suffer from discomfort and fatigue when using gesture-controlled interfaces, due to the lack of physical feedback. To solve the problem, we propose a novel complete solution of a hand gesture control system employing immersive tactile feedback to the user’s hand. For this goal, we first developed a fast and accurate hand-tracking algorithm with a Kinect sensor using the proposed MLBP (modified local binary pattern) that can efficiently analyze 3D shapes in depth images. The superiority of our tracking method was verified in terms of tracking accuracy and speed by comparing with existing methods, Natural Interaction Technology for End-user (NITE), 3D Hand Tracker and CamShift. As the second step, a new tactile feedback technology with a piezoelectric actuator has been developed and integrated into the developed hand tracking algorithm, including the DTW (dynamic time warping) gesture recognition algorithm for a complete solution of an immersive gesture control system. The quantitative and qualitative evaluations of the integrated system were conducted with human subjects, and the results demonstrate that our gesture control with tactile feedback is a promising technology compared to a vision-based gesture control system that has typically no feedback for the user’s gesture inputs. Our study provides researchers and designers with informative guidelines to develop more natural gesture control systems or immersive user interfaces with haptic feedback.",
author = "Kwangtaek Kim and Joongrock Kim and Jaesung Choi and Junghyun Kim and Sangyoun Lee",
year = "2015",
month = "1",
day = "8",
doi = "10.3390/s150101022",
language = "English",
volume = "15",
pages = "1022--1046",
journal = "Sensors",
issn = "1424-3210",
publisher = "Multidisciplinary Digital Publishing Institute (MDPI)",
number = "1",

}

Depth camera-based 3D hand gesture controls with immersive tactile feedback for natural mid-air gesture interactions. / Kim, Kwangtaek; Kim, Joongrock; Choi, Jaesung; Kim, Junghyun; Lee, Sangyoun.

In: Sensors (Switzerland), Vol. 15, No. 1, 08.01.2015, p. 1022-1046.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Depth camera-based 3D hand gesture controls with immersive tactile feedback for natural mid-air gesture interactions

AU - Kim, Kwangtaek

AU - Kim, Joongrock

AU - Choi, Jaesung

AU - Kim, Junghyun

AU - Lee, Sangyoun

PY - 2015/1/8

Y1 - 2015/1/8

N2 - Vision-based hand gesture interactions are natural and intuitive when interacting with computers, since we naturally exploit gestures to communicate with other people. However, it is agreed that users suffer from discomfort and fatigue when using gesture-controlled interfaces, due to the lack of physical feedback. To solve the problem, we propose a novel complete solution of a hand gesture control system employing immersive tactile feedback to the user’s hand. For this goal, we first developed a fast and accurate hand-tracking algorithm with a Kinect sensor using the proposed MLBP (modified local binary pattern) that can efficiently analyze 3D shapes in depth images. The superiority of our tracking method was verified in terms of tracking accuracy and speed by comparing with existing methods, Natural Interaction Technology for End-user (NITE), 3D Hand Tracker and CamShift. As the second step, a new tactile feedback technology with a piezoelectric actuator has been developed and integrated into the developed hand tracking algorithm, including the DTW (dynamic time warping) gesture recognition algorithm for a complete solution of an immersive gesture control system. The quantitative and qualitative evaluations of the integrated system were conducted with human subjects, and the results demonstrate that our gesture control with tactile feedback is a promising technology compared to a vision-based gesture control system that has typically no feedback for the user’s gesture inputs. Our study provides researchers and designers with informative guidelines to develop more natural gesture control systems or immersive user interfaces with haptic feedback.

AB - Vision-based hand gesture interactions are natural and intuitive when interacting with computers, since we naturally exploit gestures to communicate with other people. However, it is agreed that users suffer from discomfort and fatigue when using gesture-controlled interfaces, due to the lack of physical feedback. To solve the problem, we propose a novel complete solution of a hand gesture control system employing immersive tactile feedback to the user’s hand. For this goal, we first developed a fast and accurate hand-tracking algorithm with a Kinect sensor using the proposed MLBP (modified local binary pattern) that can efficiently analyze 3D shapes in depth images. The superiority of our tracking method was verified in terms of tracking accuracy and speed by comparing with existing methods, Natural Interaction Technology for End-user (NITE), 3D Hand Tracker and CamShift. As the second step, a new tactile feedback technology with a piezoelectric actuator has been developed and integrated into the developed hand tracking algorithm, including the DTW (dynamic time warping) gesture recognition algorithm for a complete solution of an immersive gesture control system. The quantitative and qualitative evaluations of the integrated system were conducted with human subjects, and the results demonstrate that our gesture control with tactile feedback is a promising technology compared to a vision-based gesture control system that has typically no feedback for the user’s gesture inputs. Our study provides researchers and designers with informative guidelines to develop more natural gesture control systems or immersive user interfaces with haptic feedback.

UR - http://www.scopus.com/inward/record.url?scp=84920931599&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84920931599&partnerID=8YFLogxK

U2 - 10.3390/s150101022

DO - 10.3390/s150101022

M3 - Article

C2 - 25580901

AN - SCOPUS:84920931599

VL - 15

SP - 1022

EP - 1046

JO - Sensors

JF - Sensors

SN - 1424-3210

IS - 1

ER -