With the rapid increase in the amount of video data, efficient object recognition is mandatory for a system capable of automatically performing question and answering. In particular, real-world video environments with numerous types of objects and complex relationships require extensive knowledge representation and inference algorithms with the properties and relations of objects. In this paper, we propose a hybrid neuro-symbolic AI system that handles scene-graph of real-world video data. The method combines neural networks that generate scene graphs in consideration of the relationship between objects on real roads and symbol-based inference algorithms for responding to questions. We define object properties, relationships, and question coverage to cover the real-world objects in pedestrian video and traverse a scene-graph to perform complex visual question-answering. We have demonstrated the superiority of the proposed method by confirming that it answered with 99.71% accuracy to 5-types of questions in a pedestrian video environment.
|Title of host publication||Hybrid Artificial Intelligent Systems - 17th International Conference, HAIS 2022, Proceedings|
|Editors||Pablo García Bringas, Hilde Pérez García, Francisco Javier Martínez de Pisón, José Ramón Villar Flecha, Alicia Troncoso Lora, Enrique A. de la Cal, Alvaro Herrero, Francisco Martínez Álvarez, Giuseppe Psaila, Hector Quintián, Emilio Corchado|
|Publisher||Springer Science and Business Media Deutschland GmbH|
|Number of pages||12|
|Publication status||Published - 2022|
|Event||17th International Conference on Hybrid Artificial Intelligence Systems, HAIS 2022 - Salamancaa, Spain|
Duration: 2022 Sept 5 → 2022 Sept 7
|Name||Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)|
|Conference||17th International Conference on Hybrid Artificial Intelligence Systems, HAIS 2022|
|Period||22/9/5 → 22/9/7|
Bibliographical noteFunding Information:
Acknowledgment. This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei University); No. 2021-0-02068, Artificial Intelligence Innovation Hub).
© 2022, Springer Nature Switzerland AG.
All Science Journal Classification (ASJC) codes
- Theoretical Computer Science
- Computer Science(all)