Semi-supervised Learning for Instrument Detection with a Class Imbalanced Dataset

Jihun Yoon; Jiwon Lee; Sung Hyun Park; Woo Jin Hyung; Min Kook Choi

doi:10.1007/978-3-030-61166-8_28

Semi-supervised Learning for Instrument Detection with a Class Imbalanced Dataset

Jihun Yoon, Jiwon Lee, Sung Hyun Park, Woo Jin Hyung, Min Kook Choi

Department of Surgery

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

Abstract

The automated recognition of surgical instruments in surgical videos is an essential factor for the evaluation and analysis of surgery. The analysis of surgical instrument localization information can help in analyses related to surgical evaluation and decision making during surgery. To solve the problem of the localization of surgical instruments, we used an object detector with bounding box labels to train the localization of the surgical tools shown in a surgical video. In this study, we propose a semi-supervised learning-based training method to solve the class imbalance between surgical instruments, which makes it challenging to train the detectors of the surgical instruments. First, we labeled gastrectomy videos for gastric cancer performed in 24 cases of robotic surgery to detect the initial bounding box of the surgical instruments. Next, a trained instrument detector was used to discern the unlabeled videos, and new labels were added to the tools causing class imbalance based on the previously acquired statistics of the labeled videos. We also performed object tracking-based label generation in the spatio-temporal domain to obtain accurate label information from the unlabeled videos in an automated manner. We were able to generate dense labels for the surgical instruments lacking labels through bidirectional object tracking using a single object tracker; thus, we achieved improved instrument detection in a fully or semi-automated manner.

Original language	English
Title of host publication	Interpretable and Annotation-Efficient Learning for Medical Image Computing - 3rd International Workshop, iMIMIC 2020, 2nd International Workshop, MIL3iD 2020, and 5th International Workshop, LABELS 2020, Held in Conjunction with MICCAI 2020, Proceedings
Editors	Jaime Cardoso, Wilson Silva, Ricardo Cruz, Hien Van Nguyen, Badri Roysam, Nicholas Heller, Pedro Henriques Abreu, Jose Pereira Amorim, Ivana Isgum, Vishal Patel, Kevin Zhou, Steve Jiang, Ngan Le, Khoa Luu, Raphael Sznitman, Veronika Cheplygina, Samaneh Abbasi, Diana Mateus, Emanuele Trucco
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	266-276
Number of pages	11
ISBN (Print)	9783030611651
DOIs	https://doi.org/10.1007/978-3-030-61166-8_28
Publication status	Published - 2020
Event	3rd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2020, the 2nd International Workshop on Medical Image Learning with Less Labels and Imperfect Data, MIL3ID 2020, and the 5th International Workshop on Large-scale Annotation of Biomedical data and Expert Label Synthesis, LABELS 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2020 - Lima, Peru Duration: 2020 Oct 4 → 2020 Oct 8

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12446 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	3rd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2020, the 2nd International Workshop on Medical Image Learning with Less Labels and Imperfect Data, MIL3ID 2020, and the 5th International Workshop on Large-scale Annotation of Biomedical data and Expert Label Synthesis, LABELS 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2020
Country/Territory	Peru
City	Lima
Period	20/10/4 → 20/10/8

Bibliographical note

Publisher Copyright:
© 2020, Springer Nature Switzerland AG.

All Science Journal Classification (ASJC) codes

Theoretical Computer Science
Computer Science(all)

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1007/978-3-030-61166-8_28

Cite this

Yoon, J., Lee, J., Park, S. H., Hyung, W. J., & Choi, M. K. (2020). Semi-supervised Learning for Instrument Detection with a Class Imbalanced Dataset. In J. Cardoso, W. Silva, R. Cruz, H. Van Nguyen, B. Roysam, N. Heller, P. Henriques Abreu, J. Pereira Amorim, I. Isgum, V. Patel, K. Zhou, S. Jiang, N. Le, K. Luu, R. Sznitman, V. Cheplygina, S. Abbasi, D. Mateus, & E. Trucco (Eds.), Interpretable and Annotation-Efficient Learning for Medical Image Computing - 3rd International Workshop, iMIMIC 2020, 2nd International Workshop, MIL3iD 2020, and 5th International Workshop, LABELS 2020, Held in Conjunction with MICCAI 2020, Proceedings (pp. 266-276). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12446 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-61166-8_28

Yoon, Jihun ; Lee, Jiwon ; Park, Sung Hyun et al. / Semi-supervised Learning for Instrument Detection with a Class Imbalanced Dataset. Interpretable and Annotation-Efficient Learning for Medical Image Computing - 3rd International Workshop, iMIMIC 2020, 2nd International Workshop, MIL3iD 2020, and 5th International Workshop, LABELS 2020, Held in Conjunction with MICCAI 2020, Proceedings. editor / Jaime Cardoso ; Wilson Silva ; Ricardo Cruz ; Hien Van Nguyen ; Badri Roysam ; Nicholas Heller ; Pedro Henriques Abreu ; Jose Pereira Amorim ; Ivana Isgum ; Vishal Patel ; Kevin Zhou ; Steve Jiang ; Ngan Le ; Khoa Luu ; Raphael Sznitman ; Veronika Cheplygina ; Samaneh Abbasi ; Diana Mateus ; Emanuele Trucco. Springer Science and Business Media Deutschland GmbH, 2020. pp. 266-276 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{6b4601b3b6714a00a535d53d928b6fdd,

title = "Semi-supervised Learning for Instrument Detection with a Class Imbalanced Dataset",

abstract = "The automated recognition of surgical instruments in surgical videos is an essential factor for the evaluation and analysis of surgery. The analysis of surgical instrument localization information can help in analyses related to surgical evaluation and decision making during surgery. To solve the problem of the localization of surgical instruments, we used an object detector with bounding box labels to train the localization of the surgical tools shown in a surgical video. In this study, we propose a semi-supervised learning-based training method to solve the class imbalance between surgical instruments, which makes it challenging to train the detectors of the surgical instruments. First, we labeled gastrectomy videos for gastric cancer performed in 24 cases of robotic surgery to detect the initial bounding box of the surgical instruments. Next, a trained instrument detector was used to discern the unlabeled videos, and new labels were added to the tools causing class imbalance based on the previously acquired statistics of the labeled videos. We also performed object tracking-based label generation in the spatio-temporal domain to obtain accurate label information from the unlabeled videos in an automated manner. We were able to generate dense labels for the surgical instruments lacking labels through bidirectional object tracking using a single object tracker; thus, we achieved improved instrument detection in a fully or semi-automated manner.",

author = "Jihun Yoon and Jiwon Lee and Park, {Sung Hyun} and Hyung, {Woo Jin} and Choi, {Min Kook}",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 3rd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2020, the 2nd International Workshop on Medical Image Learning with Less Labels and Imperfect Data, MIL3ID 2020, and the 5th International Workshop on Large-scale Annotation of Biomedical data and Expert Label Synthesis, LABELS 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2020 ; Conference date: 04-10-2020 Through 08-10-2020",

year = "2020",

doi = "10.1007/978-3-030-61166-8_28",

language = "English",

isbn = "9783030611651",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "266--276",

editor = "Jaime Cardoso and Wilson Silva and Ricardo Cruz and {Van Nguyen}, Hien and Badri Roysam and Nicholas Heller and {Henriques Abreu}, Pedro and {Pereira Amorim}, Jose and Ivana Isgum and Vishal Patel and Kevin Zhou and Steve Jiang and Ngan Le and Khoa Luu and Raphael Sznitman and Veronika Cheplygina and Samaneh Abbasi and Diana Mateus and Emanuele Trucco",

booktitle = "Interpretable and Annotation-Efficient Learning for Medical Image Computing - 3rd International Workshop, iMIMIC 2020, 2nd International Workshop, MIL3iD 2020, and 5th International Workshop, LABELS 2020, Held in Conjunction with MICCAI 2020, Proceedings",

address = "Germany",

}

Yoon, J, Lee, J, Park, SH, Hyung, WJ & Choi, MK 2020, Semi-supervised Learning for Instrument Detection with a Class Imbalanced Dataset. in J Cardoso, W Silva, R Cruz, H Van Nguyen, B Roysam, N Heller, P Henriques Abreu, J Pereira Amorim, I Isgum, V Patel, K Zhou, S Jiang, N Le, K Luu, R Sznitman, V Cheplygina, S Abbasi, D Mateus & E Trucco (eds), Interpretable and Annotation-Efficient Learning for Medical Image Computing - 3rd International Workshop, iMIMIC 2020, 2nd International Workshop, MIL3iD 2020, and 5th International Workshop, LABELS 2020, Held in Conjunction with MICCAI 2020, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12446 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 266-276, 3rd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2020, the 2nd International Workshop on Medical Image Learning with Less Labels and Imperfect Data, MIL3ID 2020, and the 5th International Workshop on Large-scale Annotation of Biomedical data and Expert Label Synthesis, LABELS 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2020, Lima, Peru, 20/10/4. https://doi.org/10.1007/978-3-030-61166-8_28

Semi-supervised Learning for Instrument Detection with a Class Imbalanced Dataset. / Yoon, Jihun; Lee, Jiwon; Park, Sung Hyun et al.

Interpretable and Annotation-Efficient Learning for Medical Image Computing - 3rd International Workshop, iMIMIC 2020, 2nd International Workshop, MIL3iD 2020, and 5th International Workshop, LABELS 2020, Held in Conjunction with MICCAI 2020, Proceedings. ed. / Jaime Cardoso; Wilson Silva; Ricardo Cruz; Hien Van Nguyen; Badri Roysam; Nicholas Heller; Pedro Henriques Abreu; Jose Pereira Amorim; Ivana Isgum; Vishal Patel; Kevin Zhou; Steve Jiang; Ngan Le; Khoa Luu; Raphael Sznitman; Veronika Cheplygina; Samaneh Abbasi; Diana Mateus; Emanuele Trucco. Springer Science and Business Media Deutschland GmbH, 2020. p. 266-276 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12446 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Semi-supervised Learning for Instrument Detection with a Class Imbalanced Dataset

AU - Yoon, Jihun

AU - Lee, Jiwon

AU - Park, Sung Hyun

AU - Hyung, Woo Jin

AU - Choi, Min Kook

PY - 2020

Y1 - 2020

N2 - The automated recognition of surgical instruments in surgical videos is an essential factor for the evaluation and analysis of surgery. The analysis of surgical instrument localization information can help in analyses related to surgical evaluation and decision making during surgery. To solve the problem of the localization of surgical instruments, we used an object detector with bounding box labels to train the localization of the surgical tools shown in a surgical video. In this study, we propose a semi-supervised learning-based training method to solve the class imbalance between surgical instruments, which makes it challenging to train the detectors of the surgical instruments. First, we labeled gastrectomy videos for gastric cancer performed in 24 cases of robotic surgery to detect the initial bounding box of the surgical instruments. Next, a trained instrument detector was used to discern the unlabeled videos, and new labels were added to the tools causing class imbalance based on the previously acquired statistics of the labeled videos. We also performed object tracking-based label generation in the spatio-temporal domain to obtain accurate label information from the unlabeled videos in an automated manner. We were able to generate dense labels for the surgical instruments lacking labels through bidirectional object tracking using a single object tracker; thus, we achieved improved instrument detection in a fully or semi-automated manner.

AB - The automated recognition of surgical instruments in surgical videos is an essential factor for the evaluation and analysis of surgery. The analysis of surgical instrument localization information can help in analyses related to surgical evaluation and decision making during surgery. To solve the problem of the localization of surgical instruments, we used an object detector with bounding box labels to train the localization of the surgical tools shown in a surgical video. In this study, we propose a semi-supervised learning-based training method to solve the class imbalance between surgical instruments, which makes it challenging to train the detectors of the surgical instruments. First, we labeled gastrectomy videos for gastric cancer performed in 24 cases of robotic surgery to detect the initial bounding box of the surgical instruments. Next, a trained instrument detector was used to discern the unlabeled videos, and new labels were added to the tools causing class imbalance based on the previously acquired statistics of the labeled videos. We also performed object tracking-based label generation in the spatio-temporal domain to obtain accurate label information from the unlabeled videos in an automated manner. We were able to generate dense labels for the surgical instruments lacking labels through bidirectional object tracking using a single object tracker; thus, we achieved improved instrument detection in a fully or semi-automated manner.

UR - http://www.scopus.com/inward/record.url?scp=85092913965&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85092913965&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-61166-8_28

DO - 10.1007/978-3-030-61166-8_28

M3 - Conference contribution

AN - SCOPUS:85092913965

SN - 9783030611651

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 266

EP - 276

BT - Interpretable and Annotation-Efficient Learning for Medical Image Computing - 3rd International Workshop, iMIMIC 2020, 2nd International Workshop, MIL3iD 2020, and 5th International Workshop, LABELS 2020, Held in Conjunction with MICCAI 2020, Proceedings

A2 - Cardoso, Jaime

A2 - Silva, Wilson

A2 - Cruz, Ricardo

A2 - Van Nguyen, Hien

A2 - Roysam, Badri

A2 - Heller, Nicholas

A2 - Henriques Abreu, Pedro

A2 - Pereira Amorim, Jose

A2 - Isgum, Ivana

A2 - Patel, Vishal

A2 - Zhou, Kevin

A2 - Jiang, Steve

A2 - Le, Ngan

A2 - Luu, Khoa

A2 - Sznitman, Raphael

A2 - Cheplygina, Veronika

A2 - Abbasi, Samaneh

A2 - Mateus, Diana

A2 - Trucco, Emanuele

PB - Springer Science and Business Media Deutschland GmbH

T2 - 3rd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2020, the 2nd International Workshop on Medical Image Learning with Less Labels and Imperfect Data, MIL3ID 2020, and the 5th International Workshop on Large-scale Annotation of Biomedical data and Expert Label Synthesis, LABELS 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2020

Y2 - 4 October 2020 through 8 October 2020

ER -

Yoon J, Lee J, Park SH, Hyung WJ, Choi MK. Semi-supervised Learning for Instrument Detection with a Class Imbalanced Dataset. In Cardoso J, Silva W, Cruz R, Van Nguyen H, Roysam B, Heller N, Henriques Abreu P, Pereira Amorim J, Isgum I, Patel V, Zhou K, Jiang S, Le N, Luu K, Sznitman R, Cheplygina V, Abbasi S, Mateus D, Trucco E, editors, Interpretable and Annotation-Efficient Learning for Medical Image Computing - 3rd International Workshop, iMIMIC 2020, 2nd International Workshop, MIL3iD 2020, and 5th International Workshop, LABELS 2020, Held in Conjunction with MICCAI 2020, Proceedings. Springer Science and Business Media Deutschland GmbH. 2020. p. 266-276. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-61166-8_28

Semi-supervised Learning for Instrument Detection with a Class Imbalanced Dataset

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this