Visual Modalities Based Multimodal Fusion for Surgical Phase Recognition

Bogyu Park; Hyeongyu Chi; Bokyung Park; Jiwon Lee; Sunghyun Park; Woo Jin Hyung; Min Kook Choi

doi:10.1007/978-3-031-18814-5_2

Visual Modalities Based Multimodal Fusion for Surgical Phase Recognition

Bogyu Park, Hyeongyu Chi, Bokyung Park, Jiwon Lee, Sunghyun Park, Woo Jin Hyung, Min Kook Choi

Department of Surgery

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We propose visual modalities-based multimodal fusion for surgical phase recognition to overcome the limitation of the diversity of information such as the presence of tools. Through the proposed methods, we extracted a visual kinematics-based index related to the usage of tools such as movement and the relation between tools in surgery. In addition, we improved recognition performance using the effective fusion method which is fusing CNN-based visual feature and visual kinematics-based index. The visual kinematics-based index is helpful for understanding the surgical procedure as the information related to the interaction between tools. Furthermore, these indices can be extracted in any environment unlike kinematics in robotic surgery. The proposed methodology was applied to two multimodal datasets to verify that it can help to improve recognition performance in clinical environments.

Original language	English
Title of host publication	Multiscale Multimodal Medical Imaging - 3rd International Workshop, MMMI 2022, Held in Conjunction with MICCAI 2022, Proceedings
Editors	Xiang Li, Quanzheng Li, Jinglei Lv, Yuankai Huo, Bin Dong, Richard M. Leahy
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	11-23
Number of pages	13
ISBN (Print)	9783031188138
DOIs	https://doi.org/10.1007/978-3-031-18814-5_2
Publication status	Published - 2022
Event	3rd International Workshop on Multiscale Multimodal Medical Imaging, MMMI 2022, held in conjunction with the 25th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2022 - Singapore, Singapore Duration: 2022 Sep 22 → 2022 Sep 22

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13594 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	3rd International Workshop on Multiscale Multimodal Medical Imaging, MMMI 2022, held in conjunction with the 25th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2022
Country/Territory	Singapore
City	Singapore
Period	22/9/22 → 22/9/22

Bibliographical note

Funding Information:
Acknowledgement. “This research was funded by the Ministry of Health & Welfare, Republic of Korea (grant number : 1465035498 / HI21C1753000022).”

Publisher Copyright:
© 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

All Science Journal Classification (ASJC) codes

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-031-18814-5_2

Cite this

Park, B., Chi, H., Park, B., Lee, J., Park, S., Hyung, W. J., & Choi, M. K. (2022). Visual Modalities Based Multimodal Fusion for Surgical Phase Recognition. In X. Li, Q. Li, J. Lv, Y. Huo, B. Dong, & R. M. Leahy (Eds.), Multiscale Multimodal Medical Imaging - 3rd International Workshop, MMMI 2022, Held in Conjunction with MICCAI 2022, Proceedings (pp. 11-23). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13594 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-18814-5_2

Park, Bogyu ; Chi, Hyeongyu ; Park, Bokyung et al. / Visual Modalities Based Multimodal Fusion for Surgical Phase Recognition. Multiscale Multimodal Medical Imaging - 3rd International Workshop, MMMI 2022, Held in Conjunction with MICCAI 2022, Proceedings. editor / Xiang Li ; Quanzheng Li ; Jinglei Lv ; Yuankai Huo ; Bin Dong ; Richard M. Leahy. Springer Science and Business Media Deutschland GmbH, 2022. pp. 11-23 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{e0f273d589bb450ea3f350e797153925,

title = "Visual Modalities Based Multimodal Fusion for Surgical Phase Recognition",

abstract = "We propose visual modalities-based multimodal fusion for surgical phase recognition to overcome the limitation of the diversity of information such as the presence of tools. Through the proposed methods, we extracted a visual kinematics-based index related to the usage of tools such as movement and the relation between tools in surgery. In addition, we improved recognition performance using the effective fusion method which is fusing CNN-based visual feature and visual kinematics-based index. The visual kinematics-based index is helpful for understanding the surgical procedure as the information related to the interaction between tools. Furthermore, these indices can be extracted in any environment unlike kinematics in robotic surgery. The proposed methodology was applied to two multimodal datasets to verify that it can help to improve recognition performance in clinical environments.",

author = "Bogyu Park and Hyeongyu Chi and Bokyung Park and Jiwon Lee and Sunghyun Park and Hyung, {Woo Jin} and Choi, {Min Kook}",

note = "Funding Information: Acknowledgement. “This research was funded by the Ministry of Health & Welfare, Republic of Korea (grant number : 1465035498 / HI21C1753000022).” Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 3rd International Workshop on Multiscale Multimodal Medical Imaging, MMMI 2022, held in conjunction with the 25th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2022 ; Conference date: 22-09-2022 Through 22-09-2022",

year = "2022",

doi = "10.1007/978-3-031-18814-5_2",

language = "English",

isbn = "9783031188138",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "11--23",

editor = "Xiang Li and Quanzheng Li and Jinglei Lv and Yuankai Huo and Bin Dong and Leahy, {Richard M.}",

booktitle = "Multiscale Multimodal Medical Imaging - 3rd International Workshop, MMMI 2022, Held in Conjunction with MICCAI 2022, Proceedings",

address = "Germany",

}

Park, B, Chi, H, Park, B, Lee, J, Park, S, Hyung, WJ & Choi, MK 2022, Visual Modalities Based Multimodal Fusion for Surgical Phase Recognition. in X Li, Q Li, J Lv, Y Huo, B Dong & RM Leahy (eds), Multiscale Multimodal Medical Imaging - 3rd International Workshop, MMMI 2022, Held in Conjunction with MICCAI 2022, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13594 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 11-23, 3rd International Workshop on Multiscale Multimodal Medical Imaging, MMMI 2022, held in conjunction with the 25th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2022, Singapore, Singapore, 22/9/22. https://doi.org/10.1007/978-3-031-18814-5_2

Visual Modalities Based Multimodal Fusion for Surgical Phase Recognition. / Park, Bogyu; Chi, Hyeongyu; Park, Bokyung et al.

Multiscale Multimodal Medical Imaging - 3rd International Workshop, MMMI 2022, Held in Conjunction with MICCAI 2022, Proceedings. ed. / Xiang Li; Quanzheng Li; Jinglei Lv; Yuankai Huo; Bin Dong; Richard M. Leahy. Springer Science and Business Media Deutschland GmbH, 2022. p. 11-23 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13594 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Visual Modalities Based Multimodal Fusion for Surgical Phase Recognition

AU - Park, Bogyu

AU - Chi, Hyeongyu

AU - Park, Bokyung

AU - Lee, Jiwon

AU - Park, Sunghyun

AU - Hyung, Woo Jin

AU - Choi, Min Kook

N1 - Funding Information: Acknowledgement. “This research was funded by the Ministry of Health & Welfare, Republic of Korea (grant number : 1465035498 / HI21C1753000022).” Publisher Copyright: © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

PY - 2022

Y1 - 2022

N2 - We propose visual modalities-based multimodal fusion for surgical phase recognition to overcome the limitation of the diversity of information such as the presence of tools. Through the proposed methods, we extracted a visual kinematics-based index related to the usage of tools such as movement and the relation between tools in surgery. In addition, we improved recognition performance using the effective fusion method which is fusing CNN-based visual feature and visual kinematics-based index. The visual kinematics-based index is helpful for understanding the surgical procedure as the information related to the interaction between tools. Furthermore, these indices can be extracted in any environment unlike kinematics in robotic surgery. The proposed methodology was applied to two multimodal datasets to verify that it can help to improve recognition performance in clinical environments.

AB - We propose visual modalities-based multimodal fusion for surgical phase recognition to overcome the limitation of the diversity of information such as the presence of tools. Through the proposed methods, we extracted a visual kinematics-based index related to the usage of tools such as movement and the relation between tools in surgery. In addition, we improved recognition performance using the effective fusion method which is fusing CNN-based visual feature and visual kinematics-based index. The visual kinematics-based index is helpful for understanding the surgical procedure as the information related to the interaction between tools. Furthermore, these indices can be extracted in any environment unlike kinematics in robotic surgery. The proposed methodology was applied to two multimodal datasets to verify that it can help to improve recognition performance in clinical environments.

UR - http://www.scopus.com/inward/record.url?scp=85141851368&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85141851368&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-18814-5_2

DO - 10.1007/978-3-031-18814-5_2

M3 - Conference contribution

AN - SCOPUS:85141851368

SN - 9783031188138

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 11

EP - 23

BT - Multiscale Multimodal Medical Imaging - 3rd International Workshop, MMMI 2022, Held in Conjunction with MICCAI 2022, Proceedings

A2 - Li, Xiang

A2 - Li, Quanzheng

A2 - Lv, Jinglei

A2 - Huo, Yuankai

A2 - Dong, Bin

A2 - Leahy, Richard M.

PB - Springer Science and Business Media Deutschland GmbH

T2 - 3rd International Workshop on Multiscale Multimodal Medical Imaging, MMMI 2022, held in conjunction with the 25th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2022

Y2 - 22 September 2022 through 22 September 2022

ER -

Park B, Chi H, Park B, Lee J, Park S, Hyung WJ et al. Visual Modalities Based Multimodal Fusion for Surgical Phase Recognition. In Li X, Li Q, Lv J, Huo Y, Dong B, Leahy RM, editors, Multiscale Multimodal Medical Imaging - 3rd International Workshop, MMMI 2022, Held in Conjunction with MICCAI 2022, Proceedings. Springer Science and Business Media Deutschland GmbH. 2022. p. 11-23. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-18814-5_2