Design of seamless multi-modal interaction framework for intelligent virtual agents in wearable mixed reality environment

Ghazanfar Ali, Hong Quan Le, Junho Kim, Seung Won Hwang, Jae In Hwang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

In this paper, we present the design of a multimodal interaction framework for intelligent virtual agents in wearable mixed reality environments, especially for interactive applications at museums, botanical gardens, and similar places. These places need engaging and no-repetitive digital content delivery to maximize user involvement. An intelligent virtual agent is a promising mode for both purposes. Premises of framework is wearable mixed reality provided by MR devices supporting spatial mapping. We envisioned a seamless interaction framework by integrating potential features of spatial mapping, virtual character animations, speech recognition, gazing, domain-specific chatbot and object recognition to enhance virtual experiences and communication between users and virtual agents. By applying a modular approach and deploying computationally intensive modules on cloud-platform, we achieved a seamless virtual experience in a device with limited resources. Human-like gaze and speech interaction with a virtual agent made it more interactive. Automated mapping of body animations with the content of a speech made it more engaging. In our tests, the virtual agents responded within 2-4 seconds after the user query. The strength of the framework is flexibility and adaptability. It can be adapted to any wearable MR device supporting spatial mapping.

Original languageEnglish
Title of host publicationProceedings of the 32nd International Conference on Computer Animation and Social Agents, CASA 2019
PublisherAssociation for Computing Machinery
Pages47-52
Number of pages6
ISBN (Electronic)9781450371599
DOIs
Publication statusPublished - 2019 Jul 1
Event32nd International Conference on Computer Animation and Social Agents, CASA 2019 - Paris, France
Duration: 2019 Jul 12019 Jul 3

Publication series

NameACM International Conference Proceeding Series

Conference

Conference32nd International Conference on Computer Animation and Social Agents, CASA 2019
CountryFrance
CityParis
Period19/7/119/7/3

All Science Journal Classification (ASJC) codes

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Design of seamless multi-modal interaction framework for intelligent virtual agents in wearable mixed reality environment'. Together they form a unique fingerprint.

  • Cite this

    Ali, G., Le, H. Q., Kim, J., Hwang, S. W., & Hwang, J. I. (2019). Design of seamless multi-modal interaction framework for intelligent virtual agents in wearable mixed reality environment. In Proceedings of the 32nd International Conference on Computer Animation and Social Agents, CASA 2019 (pp. 47-52). (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3328756.3328758