Visual Modalities Based Multimodal Fusion for Surgical Phase Recognition

Bogyu Park, Hyeongyu Chi, Bokyung Park, Jiwon Lee, Sunghyun Park, Woo Jin Hyung, Min Kook Choi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We propose visual modalities-based multimodal fusion for surgical phase recognition to overcome the limitation of the diversity of information such as the presence of tools. Through the proposed methods, we extracted a visual kinematics-based index related to the usage of tools such as movement and the relation between tools in surgery. In addition, we improved recognition performance using the effective fusion method which is fusing CNN-based visual feature and visual kinematics-based index. The visual kinematics-based index is helpful for understanding the surgical procedure as the information related to the interaction between tools. Furthermore, these indices can be extracted in any environment unlike kinematics in robotic surgery. The proposed methodology was applied to two multimodal datasets to verify that it can help to improve recognition performance in clinical environments.

Original languageEnglish
Title of host publicationMultiscale Multimodal Medical Imaging - 3rd International Workshop, MMMI 2022, Held in Conjunction with MICCAI 2022, Proceedings
EditorsXiang Li, Quanzheng Li, Jinglei Lv, Yuankai Huo, Bin Dong, Richard M. Leahy
PublisherSpringer Science and Business Media Deutschland GmbH
Pages11-23
Number of pages13
ISBN (Print)9783031188138
DOIs
Publication statusPublished - 2022
Event3rd International Workshop on Multiscale Multimodal Medical Imaging, MMMI 2022, held in conjunction with the 25th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2022 - Singapore, Singapore
Duration: 2022 Sep 222022 Sep 22

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13594 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference3rd International Workshop on Multiscale Multimodal Medical Imaging, MMMI 2022, held in conjunction with the 25th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2022
Country/TerritorySingapore
CitySingapore
Period22/9/2222/9/22

Bibliographical note

Funding Information:
Acknowledgement. “This research was funded by the Ministry of Health & Welfare, Republic of Korea (grant number : 1465035498 / HI21C1753000022).”

Publisher Copyright:
© 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'Visual Modalities Based Multimodal Fusion for Surgical Phase Recognition'. Together they form a unique fingerprint.

Cite this