Automatic Re-orientation of 3D Echocardiographic Images in Virtual Reality Using Deep Learning

Lindsay Munroe; Gina Sajith; Ei Lin; Surjava Bhattacharya; Kuberan Pushparajah; John Simpson; Julia A. Schnabel; Gavin Wheeler; Alberto Gomez; Shujie Deng

doi:10.1007/978-3-030-80432-9_14

Automatic Re-orientation of 3D Echocardiographic Images in Virtual Reality Using Deep Learning

Lindsay Munroe^*, Gina Sajith, Ei Lin, Surjava Bhattacharya, Kuberan Pushparajah, John Simpson, Julia A. Schnabel, Gavin Wheeler, Alberto Gomez, Shujie Deng

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

In 3D echocardiography (3D echo), the image orientation varies depending on the position and direction of the transducer during examination. As a result, when reviewing images the user must initially identify anatomical landmarks to understand image orientation – a potentially challenging and time-consuming task. We automated this initial step by training a deep residual neural network (ResNet) to predict the rotation required to re-orient an image to the standard apical four-chamber view). Three data pre-processing strategies were explored: 2D, 2.5D and 3D. Three different loss function strategies were investigated: classification of discrete integer angles, regression with mean absolute angle error loss, and regression with geodesic loss. We then integrated the model into a virtual reality application and aligned the re-oriented 3D echo images with a standard anatomical heart model. The deep learning strategy with the highest accuracy – 2.5D classification of discrete integer angles – achieved a mean absolute angle error on the test set of 9.0^∘. This work demonstrates the potential of artificial intelligence to support visualisation and interaction in virtual reality.

Original language	English
Title of host publication	Medical Image Understanding and Analysis - 25th Annual Conference, MIUA 2021, Proceedings
Editors	Bartłomiej W. Papież, Mohammad Yaqub, Jianbo Jiao, Ana I. Namburete, J. Alison Noble
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	177-188
Number of pages	12
ISBN (Print)	9783030804312
DOIs	https://doi.org/10.1007/978-3-030-80432-9_14
Publication status	Published - 2021
Externally published	Yes
Event	25th Annual Conference on Medical Image Understanding and Analysis, MIUA 2021 - Virtual, Online Duration: 12 Jul 2021 → 14 Jul 2021

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12722 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	25th Annual Conference on Medical Image Understanding and Analysis, MIUA 2021
City	Virtual, Online
Period	12/07/21 → 14/07/21

Bibliographical note

Funding Information:
L. Munroe—This work is independent research funded by the National Institute for Health Research (NIHRi4i, 3D Heart Project, II-LA-0716-20001, https://www.3dheart. co.uk/). This work was also supported by the Wellcome/EPSRC Centre for Medical Engineering (WT203148/Z/16/Z). Lindsay Munroe and Suryava Bhattacharya would like to acknowledge funding from the EPSRC Centre for Doctoral Training in Smart Medical Imaging (EP/S022104/1). Authors also acknowledge financial support from the Department of Health via the National Institute for Health Research (NIHR) comprehensive Biomedical Research Centre award to Guy’s and St Thomas’ NHS Foundation Trust in partnership with King’s College London and King’s College Hospital NHS Foundation Trust.

Publisher Copyright:
© 2021, Springer Nature Switzerland AG.

Keywords

3D echocardiography
Deep learning
Virtual reality

Access to Document

10.1007/978-3-030-80432-9_14

Cite this

Munroe, L., Sajith, G., Lin, E., Bhattacharya, S., Pushparajah, K., Simpson, J., Schnabel, J. A., Wheeler, G., Gomez, A., & Deng, S. (2021). Automatic Re-orientation of 3D Echocardiographic Images in Virtual Reality Using Deep Learning. In B. W. Papież, M. Yaqub, J. Jiao, A. I. Namburete, & J. A. Noble (Eds.), Medical Image Understanding and Analysis - 25th Annual Conference, MIUA 2021, Proceedings (pp. 177-188). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12722 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-80432-9_14

Munroe, Lindsay ; Sajith, Gina ; Lin, Ei et al. / Automatic Re-orientation of 3D Echocardiographic Images in Virtual Reality Using Deep Learning. Medical Image Understanding and Analysis - 25th Annual Conference, MIUA 2021, Proceedings. editor / Bartłomiej W. Papież ; Mohammad Yaqub ; Jianbo Jiao ; Ana I. Namburete ; J. Alison Noble. Springer Science and Business Media Deutschland GmbH, 2021. pp. 177-188 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{312b7040d9df47199687b81bcf9604f3,

title = "Automatic Re-orientation of 3D Echocardiographic Images in Virtual Reality Using Deep Learning",

abstract = "In 3D echocardiography (3D echo), the image orientation varies depending on the position and direction of the transducer during examination. As a result, when reviewing images the user must initially identify anatomical landmarks to understand image orientation – a potentially challenging and time-consuming task. We automated this initial step by training a deep residual neural network (ResNet) to predict the rotation required to re-orient an image to the standard apical four-chamber view). Three data pre-processing strategies were explored: 2D, 2.5D and 3D. Three different loss function strategies were investigated: classification of discrete integer angles, regression with mean absolute angle error loss, and regression with geodesic loss. We then integrated the model into a virtual reality application and aligned the re-oriented 3D echo images with a standard anatomical heart model. The deep learning strategy with the highest accuracy – 2.5D classification of discrete integer angles – achieved a mean absolute angle error on the test set of 9.0∘. This work demonstrates the potential of artificial intelligence to support visualisation and interaction in virtual reality.",

keywords = "3D echocardiography, Deep learning, Virtual reality",

author = "Lindsay Munroe and Gina Sajith and Ei Lin and Surjava Bhattacharya and Kuberan Pushparajah and John Simpson and Schnabel, {Julia A.} and Gavin Wheeler and Alberto Gomez and Shujie Deng",

note = "Funding Information: L. Munroe—This work is independent research funded by the National Institute for Health Research (NIHRi4i, 3D Heart Project, II-LA-0716-20001, https://www.3dheart. co.uk/). This work was also supported by the Wellcome/EPSRC Centre for Medical Engineering (WT203148/Z/16/Z). Lindsay Munroe and Suryava Bhattacharya would like to acknowledge funding from the EPSRC Centre for Doctoral Training in Smart Medical Imaging (EP/S022104/1). Authors also acknowledge financial support from the Department of Health via the National Institute for Health Research (NIHR) comprehensive Biomedical Research Centre award to Guy{\textquoteright}s and St Thomas{\textquoteright} NHS Foundation Trust in partnership with King{\textquoteright}s College London and King{\textquoteright}s College Hospital NHS Foundation Trust. Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.; 25th Annual Conference on Medical Image Understanding and Analysis, MIUA 2021 ; Conference date: 12-07-2021 Through 14-07-2021",

year = "2021",

doi = "10.1007/978-3-030-80432-9_14",

language = "English",

isbn = "9783030804312",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "177--188",

editor = "Papie{\.z}, {Bart{\l}omiej W.} and Mohammad Yaqub and Jianbo Jiao and Namburete, {Ana I.} and Noble, {J. Alison}",

booktitle = "Medical Image Understanding and Analysis - 25th Annual Conference, MIUA 2021, Proceedings",

}

Munroe, L, Sajith, G, Lin, E, Bhattacharya, S, Pushparajah, K, Simpson, J, Schnabel, JA, Wheeler, G, Gomez, A & Deng, S 2021, Automatic Re-orientation of 3D Echocardiographic Images in Virtual Reality Using Deep Learning. in BW Papież, M Yaqub, J Jiao, AI Namburete & JA Noble (eds), Medical Image Understanding and Analysis - 25th Annual Conference, MIUA 2021, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12722 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 177-188, 25th Annual Conference on Medical Image Understanding and Analysis, MIUA 2021, Virtual, Online, 12/07/21. https://doi.org/10.1007/978-3-030-80432-9_14

Automatic Re-orientation of 3D Echocardiographic Images in Virtual Reality Using Deep Learning. / Munroe, Lindsay; Sajith, Gina; Lin, Ei et al.
Medical Image Understanding and Analysis - 25th Annual Conference, MIUA 2021, Proceedings. ed. / Bartłomiej W. Papież; Mohammad Yaqub; Jianbo Jiao; Ana I. Namburete; J. Alison Noble. Springer Science and Business Media Deutschland GmbH, 2021. p. 177-188 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12722 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Automatic Re-orientation of 3D Echocardiographic Images in Virtual Reality Using Deep Learning

AU - Munroe, Lindsay

AU - Sajith, Gina

AU - Lin, Ei

AU - Bhattacharya, Surjava

AU - Pushparajah, Kuberan

AU - Simpson, John

AU - Schnabel, Julia A.

AU - Wheeler, Gavin

AU - Gomez, Alberto

AU - Deng, Shujie

N1 - Funding Information: L. Munroe—This work is independent research funded by the National Institute for Health Research (NIHRi4i, 3D Heart Project, II-LA-0716-20001, https://www.3dheart. co.uk/). This work was also supported by the Wellcome/EPSRC Centre for Medical Engineering (WT203148/Z/16/Z). Lindsay Munroe and Suryava Bhattacharya would like to acknowledge funding from the EPSRC Centre for Doctoral Training in Smart Medical Imaging (EP/S022104/1). Authors also acknowledge financial support from the Department of Health via the National Institute for Health Research (NIHR) comprehensive Biomedical Research Centre award to Guy’s and St Thomas’ NHS Foundation Trust in partnership with King’s College London and King’s College Hospital NHS Foundation Trust. Publisher Copyright: © 2021, Springer Nature Switzerland AG.

PY - 2021

Y1 - 2021

N2 - In 3D echocardiography (3D echo), the image orientation varies depending on the position and direction of the transducer during examination. As a result, when reviewing images the user must initially identify anatomical landmarks to understand image orientation – a potentially challenging and time-consuming task. We automated this initial step by training a deep residual neural network (ResNet) to predict the rotation required to re-orient an image to the standard apical four-chamber view). Three data pre-processing strategies were explored: 2D, 2.5D and 3D. Three different loss function strategies were investigated: classification of discrete integer angles, regression with mean absolute angle error loss, and regression with geodesic loss. We then integrated the model into a virtual reality application and aligned the re-oriented 3D echo images with a standard anatomical heart model. The deep learning strategy with the highest accuracy – 2.5D classification of discrete integer angles – achieved a mean absolute angle error on the test set of 9.0∘. This work demonstrates the potential of artificial intelligence to support visualisation and interaction in virtual reality.

AB - In 3D echocardiography (3D echo), the image orientation varies depending on the position and direction of the transducer during examination. As a result, when reviewing images the user must initially identify anatomical landmarks to understand image orientation – a potentially challenging and time-consuming task. We automated this initial step by training a deep residual neural network (ResNet) to predict the rotation required to re-orient an image to the standard apical four-chamber view). Three data pre-processing strategies were explored: 2D, 2.5D and 3D. Three different loss function strategies were investigated: classification of discrete integer angles, regression with mean absolute angle error loss, and regression with geodesic loss. We then integrated the model into a virtual reality application and aligned the re-oriented 3D echo images with a standard anatomical heart model. The deep learning strategy with the highest accuracy – 2.5D classification of discrete integer angles – achieved a mean absolute angle error on the test set of 9.0∘. This work demonstrates the potential of artificial intelligence to support visualisation and interaction in virtual reality.

KW - 3D echocardiography

KW - Deep learning

KW - Virtual reality

UR - http://www.scopus.com/inward/record.url?scp=85112241013&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-80432-9_14

DO - 10.1007/978-3-030-80432-9_14

M3 - Conference contribution

AN - SCOPUS:85112241013

SN - 9783030804312

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 177

EP - 188

BT - Medical Image Understanding and Analysis - 25th Annual Conference, MIUA 2021, Proceedings

A2 - Papież, Bartłomiej W.

A2 - Yaqub, Mohammad

A2 - Jiao, Jianbo

A2 - Namburete, Ana I.

A2 - Noble, J. Alison

PB - Springer Science and Business Media Deutschland GmbH

T2 - 25th Annual Conference on Medical Image Understanding and Analysis, MIUA 2021

Y2 - 12 July 2021 through 14 July 2021

ER -

Munroe L, Sajith G, Lin E, Bhattacharya S, Pushparajah K, Simpson J et al. Automatic Re-orientation of 3D Echocardiographic Images in Virtual Reality Using Deep Learning. In Papież BW, Yaqub M, Jiao J, Namburete AI, Noble JA, editors, Medical Image Understanding and Analysis - 25th Annual Conference, MIUA 2021, Proceedings. Springer Science and Business Media Deutschland GmbH. 2021. p. 177-188. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-80432-9_14

Automatic Re-orientation of 3D Echocardiographic Images in Virtual Reality Using Deep Learning

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this