Classifying Excavator Operations with Fusion Network of Multi-modal Deep Learning Models

Jin Young Kim, Sung Bae Cho

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Prognostics and health management (PHM) aims to offer comprehensive solutions for managing equipment health. Classifying the excavator operations plays an important role in measuring the lifetime, which is one of the tasks in PHM because the effect on the lifetime depends on the operations performed by the excavator. Several researchers have struggled with classifying the operations with either sensor or video data, but most of them have difficulties with the use of single modal data only, the surrounding environment, and the exclusive feature extraction for the data in different domains. In this paper, we propose a fusion network that classifies the excavator operations with multi-modal deep learning models. Trained are multiple classifiers with specific type of data, where feature extractors are reused to place at the front of the fusion network. The proposed fusion network combines a video-based model and a sensor-based model based on deep learning. To evaluate the performance of the proposed method, experiments are conducted with the data collected from real construction workplace. The proposed method yields the accuracy of 98.48% which is higher than conventional methods, and the multi-modal deep learning models can complement each other in terms of precision, recall, and F1-score.

Original languageEnglish
Title of host publication14th International Conference on Soft Computing Models in Industrial and Environmental Applications SOCO 2019, Proceedings
EditorsHéctor Quintián, José António Sáez Muñoz, Emilio Corchado, Francisco Martínez Álvarez, Alicia Troncoso Lora
PublisherSpringer Verlag
Pages25-34
Number of pages10
ISBN (Print)9783030200541
DOIs
Publication statusPublished - 2020 Jan 1
Event14th International Conference on Soft Computing Models in Industrial and Environmental Applications, SOCO 2019 - Seville, Spain
Duration: 2019 May 132019 May 15

Publication series

NameAdvances in Intelligent Systems and Computing
Volume950
ISSN (Print)2194-5357
ISSN (Electronic)2194-5365

Conference

Conference14th International Conference on Soft Computing Models in Industrial and Environmental Applications, SOCO 2019
CountrySpain
CitySeville
Period19/5/1319/5/15

Fingerprint

Excavators
Fusion reactions
Health
Sensors
Feature extraction
Classifiers
Deep learning
Experiments

All Science Journal Classification (ASJC) codes

  • Control and Systems Engineering
  • Computer Science(all)

Cite this

Kim, J. Y., & Cho, S. B. (2020). Classifying Excavator Operations with Fusion Network of Multi-modal Deep Learning Models. In H. Quintián, J. A. Sáez Muñoz, E. Corchado, F. Martínez Álvarez, & A. Troncoso Lora (Eds.), 14th International Conference on Soft Computing Models in Industrial and Environmental Applications SOCO 2019, Proceedings (pp. 25-34). (Advances in Intelligent Systems and Computing; Vol. 950). Springer Verlag. https://doi.org/10.1007/978-3-030-20055-8_3
Kim, Jin Young ; Cho, Sung Bae. / Classifying Excavator Operations with Fusion Network of Multi-modal Deep Learning Models. 14th International Conference on Soft Computing Models in Industrial and Environmental Applications SOCO 2019, Proceedings. editor / Héctor Quintián ; José António Sáez Muñoz ; Emilio Corchado ; Francisco Martínez Álvarez ; Alicia Troncoso Lora. Springer Verlag, 2020. pp. 25-34 (Advances in Intelligent Systems and Computing).
@inproceedings{1759622d646944268dfada8fbd01d12e,
title = "Classifying Excavator Operations with Fusion Network of Multi-modal Deep Learning Models",
abstract = "Prognostics and health management (PHM) aims to offer comprehensive solutions for managing equipment health. Classifying the excavator operations plays an important role in measuring the lifetime, which is one of the tasks in PHM because the effect on the lifetime depends on the operations performed by the excavator. Several researchers have struggled with classifying the operations with either sensor or video data, but most of them have difficulties with the use of single modal data only, the surrounding environment, and the exclusive feature extraction for the data in different domains. In this paper, we propose a fusion network that classifies the excavator operations with multi-modal deep learning models. Trained are multiple classifiers with specific type of data, where feature extractors are reused to place at the front of the fusion network. The proposed fusion network combines a video-based model and a sensor-based model based on deep learning. To evaluate the performance of the proposed method, experiments are conducted with the data collected from real construction workplace. The proposed method yields the accuracy of 98.48{\%} which is higher than conventional methods, and the multi-modal deep learning models can complement each other in terms of precision, recall, and F1-score.",
author = "Kim, {Jin Young} and Cho, {Sung Bae}",
year = "2020",
month = "1",
day = "1",
doi = "10.1007/978-3-030-20055-8_3",
language = "English",
isbn = "9783030200541",
series = "Advances in Intelligent Systems and Computing",
publisher = "Springer Verlag",
pages = "25--34",
editor = "H{\'e}ctor Quinti{\'a}n and {S{\'a}ez Mu{\~n}oz}, {Jos{\'e} Ant{\'o}nio} and Emilio Corchado and {Mart{\'i}nez {\'A}lvarez}, Francisco and {Troncoso Lora}, Alicia",
booktitle = "14th International Conference on Soft Computing Models in Industrial and Environmental Applications SOCO 2019, Proceedings",
address = "Germany",

}

Kim, JY & Cho, SB 2020, Classifying Excavator Operations with Fusion Network of Multi-modal Deep Learning Models. in H Quintián, JA Sáez Muñoz, E Corchado, F Martínez Álvarez & A Troncoso Lora (eds), 14th International Conference on Soft Computing Models in Industrial and Environmental Applications SOCO 2019, Proceedings. Advances in Intelligent Systems and Computing, vol. 950, Springer Verlag, pp. 25-34, 14th International Conference on Soft Computing Models in Industrial and Environmental Applications, SOCO 2019, Seville, Spain, 19/5/13. https://doi.org/10.1007/978-3-030-20055-8_3

Classifying Excavator Operations with Fusion Network of Multi-modal Deep Learning Models. / Kim, Jin Young; Cho, Sung Bae.

14th International Conference on Soft Computing Models in Industrial and Environmental Applications SOCO 2019, Proceedings. ed. / Héctor Quintián; José António Sáez Muñoz; Emilio Corchado; Francisco Martínez Álvarez; Alicia Troncoso Lora. Springer Verlag, 2020. p. 25-34 (Advances in Intelligent Systems and Computing; Vol. 950).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Classifying Excavator Operations with Fusion Network of Multi-modal Deep Learning Models

AU - Kim, Jin Young

AU - Cho, Sung Bae

PY - 2020/1/1

Y1 - 2020/1/1

N2 - Prognostics and health management (PHM) aims to offer comprehensive solutions for managing equipment health. Classifying the excavator operations plays an important role in measuring the lifetime, which is one of the tasks in PHM because the effect on the lifetime depends on the operations performed by the excavator. Several researchers have struggled with classifying the operations with either sensor or video data, but most of them have difficulties with the use of single modal data only, the surrounding environment, and the exclusive feature extraction for the data in different domains. In this paper, we propose a fusion network that classifies the excavator operations with multi-modal deep learning models. Trained are multiple classifiers with specific type of data, where feature extractors are reused to place at the front of the fusion network. The proposed fusion network combines a video-based model and a sensor-based model based on deep learning. To evaluate the performance of the proposed method, experiments are conducted with the data collected from real construction workplace. The proposed method yields the accuracy of 98.48% which is higher than conventional methods, and the multi-modal deep learning models can complement each other in terms of precision, recall, and F1-score.

AB - Prognostics and health management (PHM) aims to offer comprehensive solutions for managing equipment health. Classifying the excavator operations plays an important role in measuring the lifetime, which is one of the tasks in PHM because the effect on the lifetime depends on the operations performed by the excavator. Several researchers have struggled with classifying the operations with either sensor or video data, but most of them have difficulties with the use of single modal data only, the surrounding environment, and the exclusive feature extraction for the data in different domains. In this paper, we propose a fusion network that classifies the excavator operations with multi-modal deep learning models. Trained are multiple classifiers with specific type of data, where feature extractors are reused to place at the front of the fusion network. The proposed fusion network combines a video-based model and a sensor-based model based on deep learning. To evaluate the performance of the proposed method, experiments are conducted with the data collected from real construction workplace. The proposed method yields the accuracy of 98.48% which is higher than conventional methods, and the multi-modal deep learning models can complement each other in terms of precision, recall, and F1-score.

UR - http://www.scopus.com/inward/record.url?scp=85065907505&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065907505&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-20055-8_3

DO - 10.1007/978-3-030-20055-8_3

M3 - Conference contribution

AN - SCOPUS:85065907505

SN - 9783030200541

T3 - Advances in Intelligent Systems and Computing

SP - 25

EP - 34

BT - 14th International Conference on Soft Computing Models in Industrial and Environmental Applications SOCO 2019, Proceedings

A2 - Quintián, Héctor

A2 - Sáez Muñoz, José António

A2 - Corchado, Emilio

A2 - Martínez Álvarez, Francisco

A2 - Troncoso Lora, Alicia

PB - Springer Verlag

ER -

Kim JY, Cho SB. Classifying Excavator Operations with Fusion Network of Multi-modal Deep Learning Models. In Quintián H, Sáez Muñoz JA, Corchado E, Martínez Álvarez F, Troncoso Lora A, editors, 14th International Conference on Soft Computing Models in Industrial and Environmental Applications SOCO 2019, Proceedings. Springer Verlag. 2020. p. 25-34. (Advances in Intelligent Systems and Computing). https://doi.org/10.1007/978-3-030-20055-8_3