학술논문
Development and Evaluation of Machine Learning in Whole-Body Magnetic Resonance Imaging for Detecting Metastases in Patients With Lung or Colon Cancer: A Diagnostic Test Accuracy Study.
Document Type
Academic Journal
Author
Rockall AG; From the Department of Surgery and Cancer, Faculty of Medicine, Imperial College London, London, United Kingdom (A.G.R., X.L., I.L., T.D.B., N.B., G.J.W., N.S., A.L., A.H., E.O.A.); Imaging Department, Imperial College Healthcare NHS Trust, London, United Kingdom (A.G.R., T.D.B., N.B., A.S., J.B., A.F., N.S., K.W., A.L., J.R., M.S.); Imperial Clinical Trials Unit, Imperial College London, London, United Kingdom (N.J., S.S., A.T.P., J.W., X.L.); Centre for Medical Imaging, University College London, London, United Kingdom (S.P., H.S., A.P., S.T.); Department of Radiology, University College London Hospital, London, United Kingdom (S.P., H.S., A.P., S.T.); Cancer Imaging, School of Biomedical Engineering & Imaging Sciences, King's College London and Department of Radiology, Guy's & St Thomas' Hospitals NHS Foundation Trust, London, United Kingdom (V.G., C.K.-M.); Department of Biomedical Imaging and Image-Guided Therapy, Medical University of Vienna, Vienna General Hospital, Vienna, Austria (G.J.W.); Royal Marsden NHS Foundation Trust and The Institute of Cancer Research, London, United Kingdom (D.-M.K., C.M., N.T., N.S., C.K.-M., K.N.D.P.); Cancer Research UK and University College London Cancer Trials Unit, London, United Kingdom (K.R.); Faculty of Engineering, Department of Computing, Imperial College London, London, United Kingdom (Q.D., B.G.); King's Cancer Prevention Group, School of Cancer and Pharmaceutical Sciences, King's College London, London, United Kingdom (J.W.); Department of Radiology, Homerton NHS Foundation Trust, London, United Kingdom (P.B.); Paul Strickland Scanner Centre, Mount Vernon Hospital (H.S.); Department of Radiology, The Hillingdon Hospitals NHS Foundation Trust, London, United Kingdom (H.S.); Thirlestaine Breast Centre, Gloucestershire Hospitals NHS Foundation Trust, London, United Kingdom (S.V.); and Nightingale-Saunders Clinical Trials & Epidemiology Unit, King's Clinical Trials Unit, London, United Kingdom (A.T.P.).; Li X; Johnson N; Lavdas I; Santhakumaran S; Prevost AT; Punwani S; Goh V; Barwick TD; Bharwani N; Sandhu A; Sidhu H; Plumb A; Burn J; Fagan A; Wengert GJ; Koh DM; Reczko K; Dou Q; Warwick J; Liu X; Messiou C; Tunariu N; Boavida P; Soneji N; Johnston EW; Kelly-Morland C; De Paepe KN; Sokhi H; Wallitt K; Lakhani A; Russell J; Salib M; Vinnicombe S; Haq A; Aboagye EO; Taylor S; Glocker B
Source
Publisher: Lippincott Williams & Wilkins Country of Publication: United States NLM ID: 0045377 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1536-0210 (Electronic) Linking ISSN: 00209996 NLM ISO Abbreviation: Invest Radiol Subsets: MEDLINE
Subject
Language
English
Abstract
Objectives: Whole-body magnetic resonance imaging (WB-MRI) has been demonstrated to be efficient and cost-effective for cancer staging. The study aim was to develop a machine learning (ML) algorithm to improve radiologists' sensitivity and specificity for metastasis detection and reduce reading times.
Materials and Methods: A retrospective analysis of 438 prospectively collected WB-MRI scans from multicenter Streamline studies (February 2013-September 2016) was undertaken. Disease sites were manually labeled using Streamline reference standard. Whole-body MRI scans were randomly allocated to training and testing sets. A model for malignant lesion detection was developed based on convolutional neural networks and a 2-stage training strategy. The final algorithm generated lesion probability heat maps. Using a concurrent reader paradigm, 25 radiologists (18 experienced, 7 inexperienced in WB-/MRI) were randomly allocated WB-MRI scans with or without ML support to detect malignant lesions over 2 or 3 reading rounds. Reads were undertaken in the setting of a diagnostic radiology reading room between November 2019 and March 2020. Reading times were recorded by a scribe. Prespecified analysis included sensitivity, specificity, interobserver agreement, and reading time of radiology readers to detect metastases with or without ML support. Reader performance for detection of the primary tumor was also evaluated.
Results: Four hundred thirty-three evaluable WB-MRI scans were allocated to algorithm training (245) or radiology testing (50 patients with metastases, from primary 117 colon [n = 117] or lung [n = 71] cancer). Among a total 562 reads by experienced radiologists over 2 reading rounds, per-patient specificity was 86.2% (ML) and 87.7% (non-ML) (-1.5% difference; 95% confidence interval [CI], -6.4%, 3.5%; P = 0.39). Sensitivity was 66.0% (ML) and 70.0% (non-ML) (-4.0% difference; 95% CI, -13.5%, 5.5%; P = 0.344). Among 161 reads by inexperienced readers, per-patient specificity in both groups was 76.3% (0% difference; 95% CI, -15.0%, 15.0%; P = 0.613), with sensitivity of 73.3% (ML) and 60.0% (non-ML) (13.3% difference; 95% CI, -7.9%, 34.5%; P = 0.313). Per-site specificity was high (>90%) for all metastatic sites and experience levels. There was high sensitivity for the detection of primary tumors (lung cancer detection rate of 98.6% with and without ML [0.0% difference; 95% CI, -2.0%, 2.0%; P = 1.00], colon cancer detection rate of 89.0% with and 90.6% without ML [-1.7% difference; 95% CI, -5.6%, 2.2%; P = 0.65]). When combining all reads from rounds 1 and 2, reading times fell by 6.2% (95% CI, -22.8%, 10.0%) when using ML. Round 2 read-times fell by 32% (95% CI, 20.8%, 42.8%) compared with round 1. Within round 2, there was a significant decrease in read-time when using ML support, estimated as 286 seconds (or 11%) quicker ( P = 0.0281), using regression analysis to account for reader experience, read round, and tumor type. Interobserver variance suggests moderate agreement, Cohen κ = 0.64; 95% CI, 0.47, 0.81 (with ML), and Cohen κ = 0.66; 95% CI, 0.47, 0.81 (without ML).
Conclusions: There was no evidence of a significant difference in per-patient sensitivity and specificity for detecting metastases or the primary tumor using concurrent ML compared with standard WB-MRI. Radiology read-times with or without ML support fell for round 2 reads compared with round 1, suggesting that readers familiarized themselves with the study reading method. During the second reading round, there was a significant reduction in reading time when using ML support.
Competing Interests: Conflicts of interest and sources of funding: A.G.R. is part of the advisory board of RoClub. This study was funded by the National Institute of Health Research Efficacy and Mechanism Evaluation Programme (trial registration: ISRCTN 23068310). The project is supported by the Imperial College London National Institute for Health Research Biomedical Research Centre, CRUK Imperial Centre, National Institute for Health Research University College London Hospitals Biomedical Research Centre, NIHR Biomedical Research Centre, and the NIHR Clinical Research Facilities and the Royal Marsden Hospital and Institute of Cancer Research. S.T. is an NIHR senior investigator. This research has been conducted using the UK Biobank Resource.
(Copyright © 2023 The Author(s). Published by Wolters Kluwer Health, Inc.)
Materials and Methods: A retrospective analysis of 438 prospectively collected WB-MRI scans from multicenter Streamline studies (February 2013-September 2016) was undertaken. Disease sites were manually labeled using Streamline reference standard. Whole-body MRI scans were randomly allocated to training and testing sets. A model for malignant lesion detection was developed based on convolutional neural networks and a 2-stage training strategy. The final algorithm generated lesion probability heat maps. Using a concurrent reader paradigm, 25 radiologists (18 experienced, 7 inexperienced in WB-/MRI) were randomly allocated WB-MRI scans with or without ML support to detect malignant lesions over 2 or 3 reading rounds. Reads were undertaken in the setting of a diagnostic radiology reading room between November 2019 and March 2020. Reading times were recorded by a scribe. Prespecified analysis included sensitivity, specificity, interobserver agreement, and reading time of radiology readers to detect metastases with or without ML support. Reader performance for detection of the primary tumor was also evaluated.
Results: Four hundred thirty-three evaluable WB-MRI scans were allocated to algorithm training (245) or radiology testing (50 patients with metastases, from primary 117 colon [n = 117] or lung [n = 71] cancer). Among a total 562 reads by experienced radiologists over 2 reading rounds, per-patient specificity was 86.2% (ML) and 87.7% (non-ML) (-1.5% difference; 95% confidence interval [CI], -6.4%, 3.5%; P = 0.39). Sensitivity was 66.0% (ML) and 70.0% (non-ML) (-4.0% difference; 95% CI, -13.5%, 5.5%; P = 0.344). Among 161 reads by inexperienced readers, per-patient specificity in both groups was 76.3% (0% difference; 95% CI, -15.0%, 15.0%; P = 0.613), with sensitivity of 73.3% (ML) and 60.0% (non-ML) (13.3% difference; 95% CI, -7.9%, 34.5%; P = 0.313). Per-site specificity was high (>90%) for all metastatic sites and experience levels. There was high sensitivity for the detection of primary tumors (lung cancer detection rate of 98.6% with and without ML [0.0% difference; 95% CI, -2.0%, 2.0%; P = 1.00], colon cancer detection rate of 89.0% with and 90.6% without ML [-1.7% difference; 95% CI, -5.6%, 2.2%; P = 0.65]). When combining all reads from rounds 1 and 2, reading times fell by 6.2% (95% CI, -22.8%, 10.0%) when using ML. Round 2 read-times fell by 32% (95% CI, 20.8%, 42.8%) compared with round 1. Within round 2, there was a significant decrease in read-time when using ML support, estimated as 286 seconds (or 11%) quicker ( P = 0.0281), using regression analysis to account for reader experience, read round, and tumor type. Interobserver variance suggests moderate agreement, Cohen κ = 0.64; 95% CI, 0.47, 0.81 (with ML), and Cohen κ = 0.66; 95% CI, 0.47, 0.81 (without ML).
Conclusions: There was no evidence of a significant difference in per-patient sensitivity and specificity for detecting metastases or the primary tumor using concurrent ML compared with standard WB-MRI. Radiology read-times with or without ML support fell for round 2 reads compared with round 1, suggesting that readers familiarized themselves with the study reading method. During the second reading round, there was a significant reduction in reading time when using ML support.
Competing Interests: Conflicts of interest and sources of funding: A.G.R. is part of the advisory board of RoClub. This study was funded by the National Institute of Health Research Efficacy and Mechanism Evaluation Programme (trial registration: ISRCTN 23068310). The project is supported by the Imperial College London National Institute for Health Research Biomedical Research Centre, CRUK Imperial Centre, National Institute for Health Research University College London Hospitals Biomedical Research Centre, NIHR Biomedical Research Centre, and the NIHR Clinical Research Facilities and the Royal Marsden Hospital and Institute of Cancer Research. S.T. is an NIHR senior investigator. This research has been conducted using the UK Biobank Resource.
(Copyright © 2023 The Author(s). Published by Wolters Kluwer Health, Inc.)