In this paper, we present the design of a multimodal interaction framework for intelligent virtual agents in wearable mixed reality environments, especially for interactive applications at museums, botanical gardens, and similar places. These places need engaging and no-repetitive digital content delivery to maximize user involvement. An intelligent virtual agent is a promising mode for both purposes. Premises of framework is wearable mixed reality provided by MR devices supporting spatial mapping. We envisioned a seamless interaction framework by integrating potential features of spatial mapping, virtual character animations, speech recognition, gazing, domain-specific chatbot and object recognition to enhance virtual experiences and communication between users and virtual agents. By applying a modular approach and deploying computationally intensive modules on cloud-platform, we achieved a seamless virtual experience in a device with limited resources. Human-like gaze and speech interaction with a virtual agent made it more interactive. Automated mapping of body animations with the content of a speech made it more engaging. In our tests, the virtual agents responded within 2-4 seconds after the user query. The strength of the framework is flexibility and adaptability. It can be adapted to any wearable MR device supporting spatial mapping.
|Title of host publication||Proceedings of the 32nd International Conference on Computer Animation and Social Agents, CASA 2019|
|Publisher||Association for Computing Machinery|
|Number of pages||6|
|Publication status||Published - 2019 Jul 1|
|Event||32nd International Conference on Computer Animation and Social Agents, CASA 2019 - Paris, France|
Duration: 2019 Jul 1 → 2019 Jul 3
|Name||ACM International Conference Proceeding Series|
|Conference||32nd International Conference on Computer Animation and Social Agents, CASA 2019|
|Period||19/7/1 → 19/7/3|
Bibliographical noteFunding Information:
This research is supported by the Korea Creative Content Agency (KOCCA) in the Culture Technology (CT) Research and Development Program and KIST Flagship Project.
All Science Journal Classification (ASJC) codes
- Human-Computer Interaction
- Computer Vision and Pattern Recognition
- Computer Networks and Communications