A Neuro-Symbolic AI System for Visual Question Answering in Pedestrian Video Sequences

Jaeil Park, Seok Jun Bu, Sung Bae Cho

Research output: Chapter in Book/Report/Conference proceedingConference contribution


With the rapid increase in the amount of video data, efficient object recognition is mandatory for a system capable of automatically performing question and answering. In particular, real-world video environments with numerous types of objects and complex relationships require extensive knowledge representation and inference algorithms with the properties and relations of objects. In this paper, we propose a hybrid neuro-symbolic AI system that handles scene-graph of real-world video data. The method combines neural networks that generate scene graphs in consideration of the relationship between objects on real roads and symbol-based inference algorithms for responding to questions. We define object properties, relationships, and question coverage to cover the real-world objects in pedestrian video and traverse a scene-graph to perform complex visual question-answering. We have demonstrated the superiority of the proposed method by confirming that it answered with 99.71% accuracy to 5-types of questions in a pedestrian video environment.

Original languageEnglish
Title of host publicationHybrid Artificial Intelligent Systems - 17th International Conference, HAIS 2022, Proceedings
EditorsPablo García Bringas, Hilde Pérez García, Francisco Javier Martínez de Pisón, José Ramón Villar Flecha, Alicia Troncoso Lora, Enrique A. de la Cal, Alvaro Herrero, Francisco Martínez Álvarez, Giuseppe Psaila, Hector Quintián, Emilio Corchado
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages12
ISBN (Print)9783031154706
Publication statusPublished - 2022
Event17th International Conference on Hybrid Artificial Intelligence Systems, HAIS 2022 - Salamancaa, Spain
Duration: 2022 Sept 52022 Sept 7

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13469 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference17th International Conference on Hybrid Artificial Intelligence Systems, HAIS 2022

Bibliographical note

Funding Information:
Acknowledgment. This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei University); No. 2021-0-02068, Artificial Intelligence Innovation Hub).

Publisher Copyright:
© 2022, Springer Nature Switzerland AG.

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'A Neuro-Symbolic AI System for Visual Question Answering in Pedestrian Video Sequences'. Together they form a unique fingerprint.

Cite this