Hands-free user interface for vr headsets based on in situ facial gesture sensing

Jinhyuk Kim, Jaekwang Cha, Shiho Kim

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)


The typical configuration of virtual reality (VR) devices consists of a head-mounted display (HMD) and handheld controllers. As such, these units have limited utility in tasks that require hand-free operation, such as in surgical operations or assembly works in cyberspace. We propose a user interface for a VR headset based on a wearer’s facial gestures for hands-free interaction, similar to a touch interface. By sensing and recognizing the expressions associated with the in situ intentional movements of a user’s facial muscles, we define a set of commands that combine predefined facial gestures with head movements. This is achieved by utilizing six pairs of infrared (IR) photocouplers positioned at the foam interface of an HMD. We demonstrate the usability and report on the user experience as well as the performance of the proposed command set using an experimental VR game without any additional controllers. We obtained more than 99% of recognition accuracy for each facial gesture throughout the three steps of experimental tests. The proposed input interface is a cost-effective and efficient solution that facilitates hands-free user operation of a VR headset using built-in infrared photocouplers positioned in the foam interface. The proposed system recognizes facial gestures and incorporates a hands-free user interface to HMD, which is similar to the touch-screen experience of a smartphone.

Original languageEnglish
Article number7206
Pages (from-to)1-17
Number of pages17
JournalSensors (Switzerland)
Issue number24
Publication statusPublished - 2020 Dec 2

Bibliographical note

Funding Information:
Funding: This work was supported by the Institute for Information and Communications Technology Promotion (IITP) grant funded by the Korean government (MSIT) (No. 2017-0-00244, HMD Facial Expression Recognition Sensor and Cyber-interaction Interface Technology).

Publisher Copyright:
© 2020 by the authors. Licensee MDPI, Basel, Switzerland.

All Science Journal Classification (ASJC) codes

  • Analytical Chemistry
  • Biochemistry
  • Atomic and Molecular Physics, and Optics
  • Instrumentation
  • Electrical and Electronic Engineering


Dive into the research topics of 'Hands-free user interface for vr headsets based on in situ facial gesture sensing'. Together they form a unique fingerprint.

Cite this