We address the problem of 3D reconstruction from uncalibrated LiDAR point cloud and stereo images. Since the usage of each sensor alone for 3D reconstruction has weaknesses in terms of density and accuracy, we propose a deep sensor fusion framework for high-precision depth estimation. The proposed architecture consists of calibration network and depth fusion network, where both networks are designed considering the trade-off between accuracy and efficiency for mobile devices. The calibration network first corrects an initial extrinsic parameter to align the input sensor coordinate systems. The accuracy of calibration is markedly improved by formulating the calibration in the depth domain. In the depth fusion network, complementary characteristics of sparse LiDAR and dense stereo depth are then encoded in a boosting manner. Since training data for the LiDAR and stereo depth fusion are rather limited, we introduce a simple but effective approach to generate pseudo ground truth labels from the raw KITTI dataset. The experimental evaluation verifies that the proposed method outperforms current state-of-the-art methods on the KITTI benchmark. We also collect data using our proprietary multi-sensor acquisition platform and verify that the proposed method generalizes across different sensor settings and scenes.
|Number of pages||15|
|Journal||IEEE Transactions on Intelligent Transportation Systems|
|Publication status||Published - 2020 Jan|
Bibliographical noteFunding Information:
This work was supported by the Institute for Information and Communications Technology Promotion (IITP) funded by the Korean Government (MSIP) under Grant 2016-0-00197.
Manuscript received June 1, 2018; revised November 16, 2018; accepted December 21, 2018. Date of publication January 23, 2019; date of current version December 31, 2019. This work was supported by the Institute for Information and Communications Technology Promotion (IITP) funded by the Korean Government (MSIP) under Grant 2016-0-00197. The Associate Editor for this paper was H. G. Jung. (Corresponding author: Kwanghoon Sohn.) The authors are with the School of Electrical and Electronic Engineering, Yonsei University, Seoul 120-749, South Korea (e-mail: firstname.lastname@example.org; email@example.com; firstname.lastname@example.org). Digital Object Identifier 10.1109/TITS.2019.2891788
All Science Journal Classification (ASJC) codes
- Automotive Engineering
- Mechanical Engineering
- Computer Science Applications