Classifying Excavator Operations with Fusion Network of Multi-modal Deep Learning Models

Jin Young Kim, Sung Bae Cho

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Prognostics and health management (PHM) aims to offer comprehensive solutions for managing equipment health. Classifying the excavator operations plays an important role in measuring the lifetime, which is one of the tasks in PHM because the effect on the lifetime depends on the operations performed by the excavator. Several researchers have struggled with classifying the operations with either sensor or video data, but most of them have difficulties with the use of single modal data only, the surrounding environment, and the exclusive feature extraction for the data in different domains. In this paper, we propose a fusion network that classifies the excavator operations with multi-modal deep learning models. Trained are multiple classifiers with specific type of data, where feature extractors are reused to place at the front of the fusion network. The proposed fusion network combines a video-based model and a sensor-based model based on deep learning. To evaluate the performance of the proposed method, experiments are conducted with the data collected from real construction workplace. The proposed method yields the accuracy of 98.48% which is higher than conventional methods, and the multi-modal deep learning models can complement each other in terms of precision, recall, and F1-score.

Original languageEnglish
Title of host publication14th International Conference on Soft Computing Models in Industrial and Environmental Applications SOCO 2019, Proceedings
EditorsHéctor Quintián, José António Sáez Muñoz, Emilio Corchado, Francisco Martínez Álvarez, Alicia Troncoso Lora
PublisherSpringer Verlag
Pages25-34
Number of pages10
ISBN (Print)9783030200541
DOIs
Publication statusPublished - 2020 Jan 1
Event14th International Conference on Soft Computing Models in Industrial and Environmental Applications, SOCO 2019 - Seville, Spain
Duration: 2019 May 132019 May 15

Publication series

NameAdvances in Intelligent Systems and Computing
Volume950
ISSN (Print)2194-5357
ISSN (Electronic)2194-5365

Conference

Conference14th International Conference on Soft Computing Models in Industrial and Environmental Applications, SOCO 2019
CountrySpain
CitySeville
Period19/5/1319/5/15

All Science Journal Classification (ASJC) codes

  • Control and Systems Engineering
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Classifying Excavator Operations with Fusion Network of Multi-modal Deep Learning Models'. Together they form a unique fingerprint.

  • Cite this

    Kim, J. Y., & Cho, S. B. (2020). Classifying Excavator Operations with Fusion Network of Multi-modal Deep Learning Models. In H. Quintián, J. A. Sáez Muñoz, E. Corchado, F. Martínez Álvarez, & A. Troncoso Lora (Eds.), 14th International Conference on Soft Computing Models in Industrial and Environmental Applications SOCO 2019, Proceedings (pp. 25-34). (Advances in Intelligent Systems and Computing; Vol. 950). Springer Verlag. https://doi.org/10.1007/978-3-030-20055-8_3