학술논문
METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII.
Document Type
Academic Journal
Author
Kocak B; Department of Radiology, University of Health Sciences, Basaksehir Cam and Sakura City Hospital, Basaksehir, Istanbul, Turkey.; Akinci D'Antonoli T; Institute of Radiology and Nuclear Medicine, Cantonal Hospital Baselland, Liestal, Switzerland. tugba.akincidantonoli@unibas.ch.; Mercaldo N; Department of Radiology, Massachusetts General Hospital, Boston, MA, USA.; Alberich-Bayarri A; Quantitative Imaging Biomarkers in Medicine (Quibim), Valencia, Spain.; Baessler B; Department of Diagnostic and Interventional Radiology, University Hospital Würzburg, Würzburg, Germany.; Ambrosini I; Department of Translational Research, Academic Radiology, University of Pisa, Pisa, Italy.; Andreychenko AE; Laboratory for Digital Public Health Technologies, ITMO University, St. Petersburg, Russian Federation.; Bakas S; Division of Computational Pathology, Department of Pathology and Laboratory Medicine, School of Medicine, Indiana University, Indianapolis, IN, USA.; Center for Federated Learning in Precision Medicine, Indiana University, Indianapolis, IN, USA.; Beets-Tan RGH; Department of Radiology, The Netherlands Cancer Institute, Amsterdam, the Netherlands.; GROW School for Oncology and Developmental Biology, Maastricht University Medical Center, Maastricht, the Netherlands.; Institute of Regional Health Research, University of Southern Denmark, Odense, Denmark.; Bressem K; Department of Radiology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt- Universität zu Berlin, Berlin, Germany.; Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Berlin, Germany.; Buvat I; Institut Curie, Inserm, PSL University, Laboratory of Translational Imaging in Oncology, Orsay, France.; Cannella R; Section of Radiology - Department of Biomedicine, Neuroscience and Advanced Diagnostics (BiND), University of Palermo, Palermo, Italy.; Cappellini LA; Department of Biomedical Sciences, Humanitas University, Pieve Emanuele, Milan, Italy.; Cavallo AU; Division of Radiology, Istituto Dermopatico dell'Immacolata (IDI) IRCCS, Rome, Italy.; Chepelev LL; Joint Department of Medical Imaging, University Health Network, University of Toronto, Toronto, Canada.; Chu LCH; The Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins University School of Medicine, Baltimore, USA.; Demircioglu A; Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital , Essen, Germany.; deSouza NM; Division of Radiotherapy and Imaging, The Institute of Cancer Research, London, UK.; Department of Imaging, The Royal Marsden National Health Service (NHS) Foundation Trust, London, UK.; Dietzel M; Department of Radiology, University Hospital Erlangen, Erlangen, Germany.; Fanni SC; Department of Translational Research, Academic Radiology, University of Pisa, Pisa, Italy.; Fedorov A; Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.; Fournier LS; Department of Radiology, Université Paris Cité, AP-HP, Hôpital Européen Georges Pompidou, PARCC UMRS 970, INSERM, Paris, France.; Giannini V; Department of Surgical Sciences, University of Turin, Turin, Italy.; Girometti R; Institute of Radiology, Department of Medicine, University of Udine, University Hospital S. Maria della Misericordia, Udine, Italy.; Groot Lipman KBW; Department of Radiology, The Netherlands Cancer Institute, Amsterdam, the Netherlands.; GROW School for Oncology and Developmental Biology, Maastricht University Medical Center, Maastricht, the Netherlands.; Department of Thoracic Oncology, Netherlands Cancer Institute, Amsterdam, the Netherlands.; Kalarakis G; Department of Neuroradiology, Karolinska University Hospital, Stockholm, Sweden.; Department of Clinical Science, Division of Radiology, Intervention and Technology (CLINTEC), Karolinska Institutet, Stockholm, Sweden.; Department of Radiology, Medical School, University of Crete, Heraklion, Greece.; Kelly BS; Department of Radiology, St Vincent's University Hospital, Dublin, Ireland.; Insight Centre for Data Analytics, UCD, Dublin, Ireland.; School of Medicine, University College Dublin, Dublin, Ireland.; Klontzas ME; Department of Medical Imaging, University Hospital of Heraklion, Crete, Greece.; Department of Radiology, School of Medicine, University of Crete, Heraklion, Crete, Greece.; Computational Biomedicine Laboratory, Institute of Computer Science, FORTH, Heraklion, Crete, Greece.; Koh DM; Department of Radiology, Royal Marsden Hospital, Sutton, UK.; Kotter E; Department of Diagnostic and Interventional Radiology, Faculty of Medicine and Medical Center-University of Freiburg, Freiburg, Germany.; Lee HY; Department of Radiology and Center for Imaging Science, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, South Korea.; Department of Health Sciences and Technology, Samsung Advanced Institute for Health Science & Technology (SAIHST), Sungkyunkwan University, Seoul, South Korea.; Maas M; Department of Radiology & Nuclear Medicine, Amsterdam UMC Location University of Amsterdam, Meibergdreef 9, Amsterdam, the Netherlands.; Marti-Bonmati L; Medical Imaging Department and Biomedical Imaging Research Group, Hospital Universitario y Politécnico La Fe and Health Research Institute, Valencia, Spain.; Müller H; University of Applied Sciences of Western Switzerland (HES-SO Valais), Sierra, Switzerland.; Department of Radiology and Medical Informatics, University of Geneva (UniGe), Geneva, Switzerland.; Obuchowski N; Quantitative Health Sciences, Lerner Research Institute, Cleveland Clinic, Cleveland, OH, USA.; Orlhac F; Institut Curie, Inserm, PSL University, Laboratory of Translational Imaging in Oncology, Orsay, France.; Papanikolaou N; Computational Clinical Imaging Group, Centre for the Unknown, Champalimaud Foundation, Lisbon, Portugal.; Department of Radiology, Royal Marsden Hospital and The Institute of Cancer Research, London, UK.; Petrash E; Radiology department, Research Institute of Pediatric Oncology and Hematology n. a. L.A. Durnov, National Medical Research Center of Oncology n. a. N.N. Blokhin Ministry of Health of Russian Federation, Moscow, Russia.; Medical Department IRA-Labs, Moscow, Russia.; Pfaehler E; Institute for advanced simulation (IAS-8): Machine learning and data analytics, Forschungszentrum Jülich, Jülich, Germany.; Pinto Dos Santos D; Department of Radiology, University Hospital of Cologne, Cologne, Germany.; Institute for Diagnostic and Interventional Radiology, Goethe-University Frankfurt Am Main, Frankfurt, Germany.; Ponsiglione A; Department of Advanced Biomedical Sciences, University of Naples Federico II, Naples, Italy.; Sabater S; Department of Radiation Oncology, Complejo Hospitalario Universitario de Albacete, Albacete, Spain.; Sardanelli F; Department of Biomedical Sciences for Health, Università degli Studi di Milano, Milan, Italy.; Unit of Radiology, IRCCS Policlinico San Donato, San Donato Milanese, Milan, Italy.; Seeböck P; Computational Imaging Research Lab, Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Vienna, Austria.; Sijtsema NM; Department of Radiation Oncology, University Medical Center Groningen, University of Groningen, Groningen, the Netherlands.; Stanzione A; Department of Advanced Biomedical Sciences, University of Naples Federico II, Naples, Italy.; Traverso A; Department of Radiotherapy, Maastro Clinic, Maastricht, the Netherlands.; School of Medicine, Vita-Salute San Raffaele University, Milan, Italy.; Ugga L; Department of Advanced Biomedical Sciences, University of Naples Federico II, Naples, Italy.; Vallières M; Department of Computer Science, Université de Sherbrooke, Sherbrooke, Canada.; Centre de recherche du Centre hospitalier universitaire de Sherbrooke, Sherbrooke, Canada.; van Dijk LV; Department of Radiation Oncology, University Medical Center Groningen, University of Groningen, Groningen, the Netherlands.; van Griethuysen JJM; Department of Radiology, The Netherlands Cancer Institute, Amsterdam, the Netherlands.; van Hamersvelt RW; Department of Radiology, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands.; van Ooijen P; Department of Radiotherapy, University of Groningen, University Medical Center Groningen, Groningen, the Netherlands.; Vernuccio F; Section of Radiology, Department of Biomedicine, Neuroscience and Advanced Diagnosis (Bi.N.D), University of Palermo, Palermo, 90127, Italy.; Wang A; Centre for Medical Imaging & Centre for Brain Research, Faculty of Medical and Health Sciences, Auckland Bioengineering Institute, University of Auckland, Auckland, New Zealand.; Williams S; Department of Radiology, Norfolk & Norwich University Hospital, Colney Lane, Norwich, Norfolk, UK.; Witowski J; Department of Radiology, New York University Grossman School of Medicine, New York, USA.; Zhang Z; School of Information and Communication Technology, Griffith University, Nathan, Brisbane, Australia.; Zwanenburg A; National Center for Tumor Diseases (NCT/UCC), Dresden, Germany.; OncoRay - National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden - Rossendorf, Dresden, Germany.; German Cancer Research Center (DKFZ), Heidelberg, Germany.; Cuocolo R; Department of Medicine, Surgery and Dentistry, University of Salerno, Baronissi, Italy.
Source
Publisher: Springer Country of Publication: Germany NLM ID: 101532453 Publication Model: Electronic Cited Medium: Print ISSN: 1869-4101 (Print) Linking ISSN: 18694101 NLM ISO Abbreviation: Insights Imaging Subsets: PubMed not MEDLINE
Subject
Language
English
ISSN
1869-4101
Abstract
Purpose: To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies.
Methods: We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated.
Result: In total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community.
Conclusion: In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers.
Critical Relevance Statement: A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning.
Key Points: • A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol. • The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time. • METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines. • A web application has been developed to help with the calculation of the METRICS score ( https://metricsscore.github.io/metrics/METRICS.html ) and a repository created to collect feedback from the radiomics community ( https://github.com/metricsscore/metrics ).
(© 2024. The Author(s).)
Methods: We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated.
Result: In total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community.
Conclusion: In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers.
Critical Relevance Statement: A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning.
Key Points: • A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol. • The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time. • METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines. • A web application has been developed to help with the calculation of the METRICS score ( https://metricsscore.github.io/metrics/METRICS.html ) and a repository created to collect feedback from the radiomics community ( https://github.com/metricsscore/metrics ).
(© 2024. The Author(s).)