Assessing the Performance of Machine Learning Methods Trained on Public Health Observational Data: A Case Study From COVID-19

Davide Pigoli; Kieran Baker; Jobie Budd; Lorraine Butler; Harry Coppock; Sabrina Egglestone; Steven G. Gilmour; Chris Holmes; David Hurley; Radka Jersakova; Ivan Kiskin; Vasiliki Koutra; Jonathon Mellor; George Nicholson; Joe Packham; Selina Patel; Richard Payne; Stephen J. Roberts; Björn W. Schuller; Ana Tendero-Cañadas; Tracey Thornley; Alexander Titcomb

doi:10.1002/sim.10211

Assessing the Performance of Machine Learning Methods Trained on Public Health Observational Data: A Case Study From COVID-19

Davide Pigoli^*, Kieran Baker, Jobie Budd, Lorraine Butler, Harry Coppock, Sabrina Egglestone, Steven G. Gilmour, Chris Holmes, David Hurley, Radka Jersakova, Ivan Kiskin, Vasiliki Koutra, Jonathon Mellor, George Nicholson, Joe Packham, Selina Patel, Richard Payne, Stephen J. Roberts, Björn W. Schuller, Ana Tendero-CañadasTracey Thornley, Alexander Titcomb

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

Abstract

From early in the coronavirus disease 2019 (COVID-19) pandemic, there was interest in using machine learning methods to predict COVID-19 infection status based on vocal audio signals, for example, cough recordings. However, early studies had limitations in terms of data collection and of how the performances of the proposed predictive models were assessed. This article describes how these limitations have been overcome in a study carried out by the Turing-RSS Health Data Laboratory and the UK Health Security Agency. As part of the study, the UK Health Security Agency collected a dataset of acoustic recordings, SARS-CoV-2 infection status and extensive study participant meta-data. This allowed us to rigorously assess state-of-the-art machine learning techniques to predict SARS-CoV-2 infection status based on vocal audio signals. The lessons learned from this project should inform future studies on statistical evaluation methods to assess the performance of machine learning techniques for public health tasks.

Original language	English
Pages (from-to)	4861-4871
Number of pages	11
Journal	Statistics in Medicine
Volume	43
Issue number	25
DOIs	https://doi.org/10.1002/sim.10211
Publication status	Published - 10 Nov 2024

Bibliographical note

Publisher Copyright:
© 2024 The Author(s). Statistics in Medicine published by John Wiley & Sons Ltd.

Keywords

UK COVID-19 vocal audio dataset
bioacoustic markers
choice of test set
confounding
matching

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1002/sim.10211

Cite this

Pigoli, D., Baker, K., Budd, J., Butler, L., Coppock, H., Egglestone, S., Gilmour, S. G., Holmes, C., Hurley, D., Jersakova, R., Kiskin, I., Koutra, V., Mellor, J., Nicholson, G., Packham, J., Patel, S., Payne, R., Roberts, S. J., Schuller, B. W., ... Titcomb, A. (2024). Assessing the Performance of Machine Learning Methods Trained on Public Health Observational Data: A Case Study From COVID-19. Statistics in Medicine, 43(25), 4861-4871. https://doi.org/10.1002/sim.10211

@article{5c9b54f7eb1440d9a0db7e999da655d1,

title = "Assessing the Performance of Machine Learning Methods Trained on Public Health Observational Data: A Case Study From COVID-19",

abstract = "From early in the coronavirus disease 2019 (COVID-19) pandemic, there was interest in using machine learning methods to predict COVID-19 infection status based on vocal audio signals, for example, cough recordings. However, early studies had limitations in terms of data collection and of how the performances of the proposed predictive models were assessed. This article describes how these limitations have been overcome in a study carried out by the Turing-RSS Health Data Laboratory and the UK Health Security Agency. As part of the study, the UK Health Security Agency collected a dataset of acoustic recordings, SARS-CoV-2 infection status and extensive study participant meta-data. This allowed us to rigorously assess state-of-the-art machine learning techniques to predict SARS-CoV-2 infection status based on vocal audio signals. The lessons learned from this project should inform future studies on statistical evaluation methods to assess the performance of machine learning techniques for public health tasks.",

keywords = "UK COVID-19 vocal audio dataset, bioacoustic markers, choice of test set, confounding, matching",

author = "Davide Pigoli and Kieran Baker and Jobie Budd and Lorraine Butler and Harry Coppock and Sabrina Egglestone and Gilmour, {Steven G.} and Chris Holmes and David Hurley and Radka Jersakova and Ivan Kiskin and Vasiliki Koutra and Jonathon Mellor and George Nicholson and Joe Packham and Selina Patel and Richard Payne and Roberts, {Stephen J.} and Schuller, {Bj{\"o}rn W.} and Ana Tendero-Ca{\~n}adas and Tracey Thornley and Alexander Titcomb",

note = "Publisher Copyright: {\textcopyright} 2024 The Author(s). Statistics in Medicine published by John Wiley & Sons Ltd.",

year = "2024",

month = nov,

day = "10",

doi = "10.1002/sim.10211",

language = "English",

volume = "43",

pages = "4861--4871",

journal = "Statistics in Medicine",

issn = "0277-6715",

publisher = "John Wiley and Sons Ltd",

number = "25",

}

Pigoli, D, Baker, K, Budd, J, Butler, L, Coppock, H, Egglestone, S, Gilmour, SG, Holmes, C, Hurley, D, Jersakova, R, Kiskin, I, Koutra, V, Mellor, J, Nicholson, G, Packham, J, Patel, S, Payne, R, Roberts, SJ, Schuller, BW, Tendero-Cañadas, A, Thornley, T & Titcomb, A 2024, 'Assessing the Performance of Machine Learning Methods Trained on Public Health Observational Data: A Case Study From COVID-19', Statistics in Medicine, vol. 43, no. 25, pp. 4861-4871. https://doi.org/10.1002/sim.10211

TY - JOUR

T1 - Assessing the Performance of Machine Learning Methods Trained on Public Health Observational Data

T2 - A Case Study From COVID-19

AU - Pigoli, Davide

AU - Baker, Kieran

AU - Budd, Jobie

AU - Butler, Lorraine

AU - Coppock, Harry

AU - Egglestone, Sabrina

AU - Gilmour, Steven G.

AU - Holmes, Chris

AU - Hurley, David

AU - Jersakova, Radka

AU - Kiskin, Ivan

AU - Koutra, Vasiliki

AU - Mellor, Jonathon

AU - Nicholson, George

AU - Packham, Joe

AU - Patel, Selina

AU - Payne, Richard

AU - Roberts, Stephen J.

AU - Schuller, Björn W.

AU - Tendero-Cañadas, Ana

AU - Thornley, Tracey

AU - Titcomb, Alexander

PY - 2024/11/10

Y1 - 2024/11/10

N2 - From early in the coronavirus disease 2019 (COVID-19) pandemic, there was interest in using machine learning methods to predict COVID-19 infection status based on vocal audio signals, for example, cough recordings. However, early studies had limitations in terms of data collection and of how the performances of the proposed predictive models were assessed. This article describes how these limitations have been overcome in a study carried out by the Turing-RSS Health Data Laboratory and the UK Health Security Agency. As part of the study, the UK Health Security Agency collected a dataset of acoustic recordings, SARS-CoV-2 infection status and extensive study participant meta-data. This allowed us to rigorously assess state-of-the-art machine learning techniques to predict SARS-CoV-2 infection status based on vocal audio signals. The lessons learned from this project should inform future studies on statistical evaluation methods to assess the performance of machine learning techniques for public health tasks.

AB - From early in the coronavirus disease 2019 (COVID-19) pandemic, there was interest in using machine learning methods to predict COVID-19 infection status based on vocal audio signals, for example, cough recordings. However, early studies had limitations in terms of data collection and of how the performances of the proposed predictive models were assessed. This article describes how these limitations have been overcome in a study carried out by the Turing-RSS Health Data Laboratory and the UK Health Security Agency. As part of the study, the UK Health Security Agency collected a dataset of acoustic recordings, SARS-CoV-2 infection status and extensive study participant meta-data. This allowed us to rigorously assess state-of-the-art machine learning techniques to predict SARS-CoV-2 infection status based on vocal audio signals. The lessons learned from this project should inform future studies on statistical evaluation methods to assess the performance of machine learning techniques for public health tasks.

KW - UK COVID-19 vocal audio dataset

KW - bioacoustic markers

KW - choice of test set

KW - confounding

KW - matching

UR - http://www.scopus.com/inward/record.url?scp=85203251335&partnerID=8YFLogxK

U2 - 10.1002/sim.10211

DO - 10.1002/sim.10211

M3 - Article

AN - SCOPUS:85203251335

SN - 0277-6715

VL - 43

SP - 4861

EP - 4871

JO - Statistics in Medicine

JF - Statistics in Medicine

IS - 25

ER -

Assessing the Performance of Machine Learning Methods Trained on Public Health Observational Data: A Case Study From COVID-19

Abstract

Bibliographical note

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this