Automated quality control in image segmentation: Application to the UK Biobank cardiovascular magnetic resonance imaging study

Robert Robinson, Vanya V. Valindria, Wenjia Bai, Ozan Oktay, Bernhard Kainz, Hideaki Suzuki, Mihir M. Sanghvi, Nay Aung, José Miguel Paiva, Filip Zemrak, Kenneth Fung, Elena Lukaschuk, Aaron M. Lee, Valentina Carapella, Young Jin Kim, Stefan K. Piechnik, Stefan Neubauer, Steffen E. Petersen, Chris Page, Paul M. MatthewsDaniel Rueckert, Ben Glocker

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

Background: The trend towards large-scale studies including population imaging poses new challenges in terms of quality control (QC). This is a particular issue when automatic processing tools such as image segmentation methods are employed to derive quantitative measures or biomarkers for further analyses. Manual inspection and visual QC of each segmentation result is not feasible at large scale. However, it is important to be able to automatically detect when a segmentation method fails in order to avoid inclusion of wrong measurements into subsequent analyses which could otherwise lead to incorrect conclusions. Methods: To overcome this challenge, we explore an approach for predicting segmentation quality based on Reverse Classification Accuracy, which enables us to discriminate between successful and failed segmentations on a per-cases basis. We validate this approach on a new, large-scale manually-annotated set of 4800 cardiovascular magnetic resonance (CMR) scans. We then apply our method to a large cohort of 7250 CMR on which we have performed manual QC. Results: We report results used for predicting segmentation quality metrics including Dice Similarity Coefficient (DSC) and surface-distance measures. As initial validation, we present data for 400 scans demonstrating 99% accuracy for classifying low and high quality segmentations using the predicted DSC scores. As further validation we show high correlation between real and predicted scores and 95% classification accuracy on 4800 scans for which manual segmentations were available. We mimic real-world application of the method on 7250 CMR where we show good agreement between predicted quality metrics and manual visual QC scores. Conclusions: We show that Reverse classification accuracy has the potential for accurate and fully automatic segmentation QC on a per-case basis in the context of large-scale population imaging as in the UK Biobank Imaging Study.

Original languageEnglish
Article number18
JournalJournal of Cardiovascular Magnetic Resonance
Volume21
Issue number1
DOIs
Publication statusPublished - 2019 Mar 14

Fingerprint

Quality Control
Magnetic Resonance Imaging
Magnetic Resonance Spectroscopy
Population
Biomarkers

All Science Journal Classification (ASJC) codes

  • Radiological and Ultrasound Technology
  • Radiology Nuclear Medicine and imaging
  • Cardiology and Cardiovascular Medicine
  • Family Practice

Cite this

Robinson, Robert ; Valindria, Vanya V. ; Bai, Wenjia ; Oktay, Ozan ; Kainz, Bernhard ; Suzuki, Hideaki ; Sanghvi, Mihir M. ; Aung, Nay ; Paiva, José Miguel ; Zemrak, Filip ; Fung, Kenneth ; Lukaschuk, Elena ; Lee, Aaron M. ; Carapella, Valentina ; Kim, Young Jin ; Piechnik, Stefan K. ; Neubauer, Stefan ; Petersen, Steffen E. ; Page, Chris ; Matthews, Paul M. ; Rueckert, Daniel ; Glocker, Ben. / Automated quality control in image segmentation : Application to the UK Biobank cardiovascular magnetic resonance imaging study. In: Journal of Cardiovascular Magnetic Resonance. 2019 ; Vol. 21, No. 1.
@article{9b755ef82e97459f83571c40c6566545,
title = "Automated quality control in image segmentation: Application to the UK Biobank cardiovascular magnetic resonance imaging study",
abstract = "Background: The trend towards large-scale studies including population imaging poses new challenges in terms of quality control (QC). This is a particular issue when automatic processing tools such as image segmentation methods are employed to derive quantitative measures or biomarkers for further analyses. Manual inspection and visual QC of each segmentation result is not feasible at large scale. However, it is important to be able to automatically detect when a segmentation method fails in order to avoid inclusion of wrong measurements into subsequent analyses which could otherwise lead to incorrect conclusions. Methods: To overcome this challenge, we explore an approach for predicting segmentation quality based on Reverse Classification Accuracy, which enables us to discriminate between successful and failed segmentations on a per-cases basis. We validate this approach on a new, large-scale manually-annotated set of 4800 cardiovascular magnetic resonance (CMR) scans. We then apply our method to a large cohort of 7250 CMR on which we have performed manual QC. Results: We report results used for predicting segmentation quality metrics including Dice Similarity Coefficient (DSC) and surface-distance measures. As initial validation, we present data for 400 scans demonstrating 99{\%} accuracy for classifying low and high quality segmentations using the predicted DSC scores. As further validation we show high correlation between real and predicted scores and 95{\%} classification accuracy on 4800 scans for which manual segmentations were available. We mimic real-world application of the method on 7250 CMR where we show good agreement between predicted quality metrics and manual visual QC scores. Conclusions: We show that Reverse classification accuracy has the potential for accurate and fully automatic segmentation QC on a per-case basis in the context of large-scale population imaging as in the UK Biobank Imaging Study.",
author = "Robert Robinson and Valindria, {Vanya V.} and Wenjia Bai and Ozan Oktay and Bernhard Kainz and Hideaki Suzuki and Sanghvi, {Mihir M.} and Nay Aung and Paiva, {Jos{\'e} Miguel} and Filip Zemrak and Kenneth Fung and Elena Lukaschuk and Lee, {Aaron M.} and Valentina Carapella and Kim, {Young Jin} and Piechnik, {Stefan K.} and Stefan Neubauer and Petersen, {Steffen E.} and Chris Page and Matthews, {Paul M.} and Daniel Rueckert and Ben Glocker",
year = "2019",
month = "3",
day = "14",
doi = "10.1186/s12968-019-0523-x",
language = "English",
volume = "21",
journal = "Journal of Cardiovascular Magnetic Resonance",
issn = "1097-6647",
publisher = "BioMed Central",
number = "1",

}

Robinson, R, Valindria, VV, Bai, W, Oktay, O, Kainz, B, Suzuki, H, Sanghvi, MM, Aung, N, Paiva, JM, Zemrak, F, Fung, K, Lukaschuk, E, Lee, AM, Carapella, V, Kim, YJ, Piechnik, SK, Neubauer, S, Petersen, SE, Page, C, Matthews, PM, Rueckert, D & Glocker, B 2019, 'Automated quality control in image segmentation: Application to the UK Biobank cardiovascular magnetic resonance imaging study', Journal of Cardiovascular Magnetic Resonance, vol. 21, no. 1, 18. https://doi.org/10.1186/s12968-019-0523-x

Automated quality control in image segmentation : Application to the UK Biobank cardiovascular magnetic resonance imaging study. / Robinson, Robert; Valindria, Vanya V.; Bai, Wenjia; Oktay, Ozan; Kainz, Bernhard; Suzuki, Hideaki; Sanghvi, Mihir M.; Aung, Nay; Paiva, José Miguel; Zemrak, Filip; Fung, Kenneth; Lukaschuk, Elena; Lee, Aaron M.; Carapella, Valentina; Kim, Young Jin; Piechnik, Stefan K.; Neubauer, Stefan; Petersen, Steffen E.; Page, Chris; Matthews, Paul M.; Rueckert, Daniel; Glocker, Ben.

In: Journal of Cardiovascular Magnetic Resonance, Vol. 21, No. 1, 18, 14.03.2019.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Automated quality control in image segmentation

T2 - Application to the UK Biobank cardiovascular magnetic resonance imaging study

AU - Robinson, Robert

AU - Valindria, Vanya V.

AU - Bai, Wenjia

AU - Oktay, Ozan

AU - Kainz, Bernhard

AU - Suzuki, Hideaki

AU - Sanghvi, Mihir M.

AU - Aung, Nay

AU - Paiva, José Miguel

AU - Zemrak, Filip

AU - Fung, Kenneth

AU - Lukaschuk, Elena

AU - Lee, Aaron M.

AU - Carapella, Valentina

AU - Kim, Young Jin

AU - Piechnik, Stefan K.

AU - Neubauer, Stefan

AU - Petersen, Steffen E.

AU - Page, Chris

AU - Matthews, Paul M.

AU - Rueckert, Daniel

AU - Glocker, Ben

PY - 2019/3/14

Y1 - 2019/3/14

N2 - Background: The trend towards large-scale studies including population imaging poses new challenges in terms of quality control (QC). This is a particular issue when automatic processing tools such as image segmentation methods are employed to derive quantitative measures or biomarkers for further analyses. Manual inspection and visual QC of each segmentation result is not feasible at large scale. However, it is important to be able to automatically detect when a segmentation method fails in order to avoid inclusion of wrong measurements into subsequent analyses which could otherwise lead to incorrect conclusions. Methods: To overcome this challenge, we explore an approach for predicting segmentation quality based on Reverse Classification Accuracy, which enables us to discriminate between successful and failed segmentations on a per-cases basis. We validate this approach on a new, large-scale manually-annotated set of 4800 cardiovascular magnetic resonance (CMR) scans. We then apply our method to a large cohort of 7250 CMR on which we have performed manual QC. Results: We report results used for predicting segmentation quality metrics including Dice Similarity Coefficient (DSC) and surface-distance measures. As initial validation, we present data for 400 scans demonstrating 99% accuracy for classifying low and high quality segmentations using the predicted DSC scores. As further validation we show high correlation between real and predicted scores and 95% classification accuracy on 4800 scans for which manual segmentations were available. We mimic real-world application of the method on 7250 CMR where we show good agreement between predicted quality metrics and manual visual QC scores. Conclusions: We show that Reverse classification accuracy has the potential for accurate and fully automatic segmentation QC on a per-case basis in the context of large-scale population imaging as in the UK Biobank Imaging Study.

AB - Background: The trend towards large-scale studies including population imaging poses new challenges in terms of quality control (QC). This is a particular issue when automatic processing tools such as image segmentation methods are employed to derive quantitative measures or biomarkers for further analyses. Manual inspection and visual QC of each segmentation result is not feasible at large scale. However, it is important to be able to automatically detect when a segmentation method fails in order to avoid inclusion of wrong measurements into subsequent analyses which could otherwise lead to incorrect conclusions. Methods: To overcome this challenge, we explore an approach for predicting segmentation quality based on Reverse Classification Accuracy, which enables us to discriminate between successful and failed segmentations on a per-cases basis. We validate this approach on a new, large-scale manually-annotated set of 4800 cardiovascular magnetic resonance (CMR) scans. We then apply our method to a large cohort of 7250 CMR on which we have performed manual QC. Results: We report results used for predicting segmentation quality metrics including Dice Similarity Coefficient (DSC) and surface-distance measures. As initial validation, we present data for 400 scans demonstrating 99% accuracy for classifying low and high quality segmentations using the predicted DSC scores. As further validation we show high correlation between real and predicted scores and 95% classification accuracy on 4800 scans for which manual segmentations were available. We mimic real-world application of the method on 7250 CMR where we show good agreement between predicted quality metrics and manual visual QC scores. Conclusions: We show that Reverse classification accuracy has the potential for accurate and fully automatic segmentation QC on a per-case basis in the context of large-scale population imaging as in the UK Biobank Imaging Study.

UR - http://www.scopus.com/inward/record.url?scp=85062837836&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062837836&partnerID=8YFLogxK

U2 - 10.1186/s12968-019-0523-x

DO - 10.1186/s12968-019-0523-x

M3 - Article

C2 - 30866968

AN - SCOPUS:85062837836

VL - 21

JO - Journal of Cardiovascular Magnetic Resonance

JF - Journal of Cardiovascular Magnetic Resonance

SN - 1097-6647

IS - 1

M1 - 18

ER -