Achieving Real-Time Sign Language Translation Using a Smartphone's True Depth Images

Hyeon Jung Park, Jong Seok Lee, Jeong Gil Ko

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Sign language is used as a visual form of communication among the deaf and is considered as an official language in many countries. While there has been many efforts to achieve efficient translation between sign and verbal languages, many of these previous work can be applied in non-mobile context or exploit RGB images which can potentially invade the users' privacy. This work presents our preliminary efforts in designing a mobile device-based sign language translation system using depth-only images. Our system performs image processing on the smartphone-collected depth images to emphasize the subject's hand and upper body gestures and exploits a convolutional neural network for feature extraction. The series of features gathered from word-representing videos are passed through a Long-Short Term Memory (LSTM) model for word-level sign language translation. We train and test our system using a total of 2,200 samples collected from 26 people for 17 words. The classification accuracy of our proposed system using the self-collected data achieves 92% with an efficient image preprocessing phase.

Original languageEnglish
Title of host publication2020 International Conference on COMmunication Systems and NETworkS, COMSNETS 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages622-625
Number of pages4
ISBN (Electronic)9781728131870
DOIs
Publication statusPublished - 2020 Jan
Event2020 International Conference on COMmunication Systems and NETworkS, COMSNETS 2020 - Bengaluru, India
Duration: 2020 Jan 72020 Jan 11

Publication series

Name2020 International Conference on COMmunication Systems and NETworkS, COMSNETS 2020

Conference

Conference2020 International Conference on COMmunication Systems and NETworkS, COMSNETS 2020
CountryIndia
CityBengaluru
Period20/1/720/1/11

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Signal Processing

Fingerprint Dive into the research topics of 'Achieving Real-Time Sign Language Translation Using a Smartphone's True Depth Images'. Together they form a unique fingerprint.

  • Cite this

    Park, H. J., Lee, J. S., & Ko, J. G. (2020). Achieving Real-Time Sign Language Translation Using a Smartphone's True Depth Images. In 2020 International Conference on COMmunication Systems and NETworkS, COMSNETS 2020 (pp. 622-625). [9027420] (2020 International Conference on COMmunication Systems and NETworkS, COMSNETS 2020). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/COMSNETS48256.2020.9027420