Discordant bioinformatic predictions of antimicrobial resistance from whole-genome sequencing data of bacterial isolates: An inter-laboratory study

Ronan M. Doyle*, Denise M. O'sullivan, Sean D. Aller, Sebastian Bruchmann, Taane Clark, Andreu Coello Pelegrin, Martin Cormican, Ernest Diez Benavente, Matthew J. Ellington, Elaine McGrath, Yair Motro, Thi Phuong Thuy Nguyen, Jody Phelan, Liam P. Shaw, Richard A. Stabler, Alex van Belkum, Lucy van Dorp, Neil Woodford, Jacob Moran-Gilad, Jim F. HuggettKathryn A. Harris

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

69 Citations (Scopus)
13 Downloads (Pure)


Antimicrobial resistance (AMR) poses a threat to public health. Clinical microbiology laboratories typically rely on culturing bacteria for antimicrobial-susceptibility testing (AST). As the implementation costs and technical barriers fall, whole-genome sequencing (WGS) has emerged as a ‘one-stop’ test for epidemiological and predictive AST results. Few published comparisons exist for the myriad analytical pipelines used for predicting AMR. To address this, we performed an inter-laboratory study providing sets of participating researchers with identical short-read WGS data from clinical isolates, allowing us to assess the reproducibility of the bioinformatic prediction of AMR between participants, and identify problem cases and factors that lead to discordant results. We produced ten WGS datasets of varying quality from cultured carbapenem-resistant organisms obtained from clinical samples sequenced on either an Illumina NextSeq or HiSeq instrument. Nine participating teams (‘participants’) were provided these sequence data without any other contextual information. Each participant used their choice of pipeline to determine the species, the presence of resistance-associated genes, and to predict susceptibility or resistance to amikacin, gentamicin, ciprofloxacin and cefotaxime. We found participants predicted different numbers of AMR-associated genes and different gene variants from the same clinical samples. The quality of the sequence data, choice of bioinformatic pipeline and interpretation of the results all contributed to discordance between participants. Although much of the inaccurate gene variant annotation did not affect genotypic resistance predictions, we observed low specificity when compared to phenotypic AST results, but this improved in samples with higher read depths. Had the results been used to predict AST and guide treatment, a different antibiotic would have been recommended for each isolate by at least one participant. These challenges, at the final analytical stage of using WGS to predict AMR, suggest the need for refinements when using this technology in clinical settings. Comprehensive public resistance sequence databases, full recommendations on sequence data quality and standardization in the comparisons between genotype and resistance phenotypes will all play a fundamental role in the successful implementation of AST prediction using WGS in clinical microbiology laboratories.

Original languageEnglish
Article number000335
JournalMicrobial Genomics
Issue number2
Publication statusPublished - 12 Feb 2020

Bibliographical note

Funding Information: This work was supported by the UK National Measurement System and the European Metrology Programme for Innovation and Research (EMPIR) joint research project (HLT07) ‘AntiMicroResist’, which has received funding from the EMPIR programme co-financed by the participating states and the European Union’s Horizon 2020 research and innovation programme. A.C.P. received funding from the European Union’s Horizon 2020 research and innovation programme ‘New Diagnostics for Infectious Diseases’ (ND4ID) under the Marie Skłodowska-Curie grant agreement no. 675412. These funding bodies had no influence on the design of the study, collection, analysis and interpretation of data, nor the writing of the manuscript.

A.C.P. and A.V.B. are employees of bioMérieux, a company developing, marketing and selling tests in the infectious disease domain. The company had no influence on the design and execution of the clinical study, neither did the company influence the choice of the diagnostic tools used during the clinical study. The opinions expressed in the manuscript are the authors', which do not necessarily reflect company policies. M.J.E. and N.W. are members of Public Health England’s AMRHAI Reference Unit, which has received financial support for conference attendance, lectures, research projects or contracted evaluations from numerous sources, including: Accelerate Diagnostics, Achaogen Inc., Allecra Therapeutics, Amplex, AstraZeneca UK Ltd, AusDiagnostics, Basilea Pharmaceutica, Becton Dickinson Diagnostics, bioMérieux, Bio-Rad Laboratories, British Society for Antimicrobial Chemotherapy, Cepheid, Check-Points B.V., Cubist Pharmaceuticals, the Department of Health, Enigma Diagnostics, European Centre for Disease Prevention and Control, Food Standards Agency, GlaxoSmithKline Services Ltd, Helperby Therapeutics, Henry Stewart Talks, IHMA Ltd, Innovate UK, Kalidex Pharmaceuticals, Melinta Therapeutics, Merck Sharpe and Dohme Corp., Meiji Seika Pharma Co. Ltd, Mobidiag, Momentum Biosciences Ltd, Neem Biotech, National Institute for Health Research, Nordic Pharma Ltd, Norgine Pharmaceuticals, Rempex Pharmaceuticals Ltd, Roche, Rokitan Ltd, Smith and Nephew UK Ltd, Shionogi and Co. Ltd, Trius Therapeutics, VenatoRx Pharmaceuticals, Wockhardt Ltd and the World Health Organization. All other authors declare that they have no competing interests and have performed the work in an individual capacity.

Publisher Copyright: © 2020 The Authors.

Citation: Doyle, Ronan M., et al. "Discordant bioinformatic predictions of antimicrobial resistance from whole-genome sequencing data of bacterial isolates: an inter-laboratory study." Microbial genomics 6.2 (2020).

DOI: https://doi.org/10.1099/mgen.0.000335


  • Antimicrobial resistance
  • Antimicrobial-susceptibility testing
  • Bioinformatics
  • Carbapenem resistance
  • Whole-genome sequencing


Dive into the research topics of 'Discordant bioinformatic predictions of antimicrobial resistance from whole-genome sequencing data of bacterial isolates: An inter-laboratory study'. Together they form a unique fingerprint.

Cite this