학술논문
Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study
Document Type
Academic Journal
Author
Thoma, Brent; Sebok-Syer, Stefanie S.; Krishnan, Keeth; Siemens, Marshall; Trueger, Seth N.; Colmers-Gray, Isabelle; Woods, Rob; Petrusa, Emil; Chan, Teresa; Alexander, Charlotte; Alkhalifah, Mohammed; Alqahtani, Saeed; Anderson, Scott; Anderson, Shelaina; Andrews, Colin; Andruko, Jocelyn; Ankel, Felix; Antony, Nikytha; Aryal, Diptesh; Backus, Barbra; Baird, Jennifer; Baker, Andrew; Batty, Sarah; Baylis, Jared; Beaumont, Braeden; Belcher, Chris; Benavides, Brent; Benham, Michael; Pelletier, Elyse Berger; Botta, Julian; Bouchard, Nicholas; Brazil, Victoria; Brumfield, Emily; Bryson, Anthony; Bunchit, Wisarut; Butler, Kat; Buzikievich, Lindy; Calcara, David; Carey, Rob; Carroll, Stephen; Lyons, Casey; Cassidy, Louise; Challen, Kirsty; Chaplin, Tim; Chatham-Zvelebil, Natasha; Chen, Eric; Chen, Lucy; Chhabra, Sushant; Chin, Alvin; Chochi, Eric; Choudhri, Tina; Christensen, Jeremy; Colmers-Gray, Isabelle; Connors, Kimberly; Coppersmith, Veronica; Cosgrove, Abby; Costello, Gregory; Cullison, Kevin; DʼAlessandro, Andrew; de Wit, Kerstin; Decock, Marie; Delbani, Rayan; Denq, William; Deutscher, Julianna; Devine, Brendan; Dorsett, Maia; Duda, Taylor; Dueweke, Justin; Dunphy, Teresa; Dyer, Sean; Eastley, Kathryn T.; Edmonds, Marcia; Edwards, Ken; Ehrman, Robert; Elkhalidy, Youness; Fedor, Preston; Ficiur, Brian; Flynn, Caley; Fraser, Bill; Fu, Meagan; Fukakusa, James; Funk, Eric; Gaco, Damjan; Gawlik, Viktor; Ghaffarian, Kenn; Gharahbaghian, Laleh; Griffith, Phil; Griffith, Andrew; Grock, Andrew; Gronowski, Tanner; Grossman, Cathy; Gucwa, Jaroslaw; Gupta, Pawan; Gustafson, Alexandra; Guy, Andrew; Haas, Mary; Haciski, Stanislaw; Hajdinjak, Emina; Hall, Andrew K.; Hammock, Regina; Hansel, Jan; Hart, Alexander; Hattin, Larissa; Herb, Brandon; Hilbert, SueLin; Hill, Jesse; Hill, Jeff; Ho, Amy; House, Emily; House, Nina; Huffman, James; Inboriboon, Charlie; Ireland, Alex; Jamal, Mohammed Ali; Jansen, Victor; Jarou, Zach; Jia, Vivian; Johnston, Levi; Kalnow, Drew; Kapur, Puneet; Kelly, Seth; Kelson, Kyle; Kent, William; Khakhkhar, Rishi; Khurana, Jaasmit; Kilp, Ashley; Knapp, Scott; Kohler, Sebastian; Kruhlak, Ivanna; Lalani, Nadim; Lam, Samantha; Lank, Patrick; Laurie, Zander; Lea, Kristina; Leber, Ernest; Lee, Ching-Hsing; Lenes, Haakon; Lenora, Nilantha; Leontowicz, Jesse; Lien, Kelly; Lin, Yingchun; Lin, Michelle; Little, Andrew; Liu, Ivy; Liu, Harry; Liu, Steve; Louka, Stephanie; Lovell, Elise; Lowe, David; Lubberdink, Ashley; Luc, Jessica; Ma, Sheng-Hsiang; MacLeod, Hugh; Mancuso, Nick; Maneshi, Anali; Carrillo, Dra. Maria Rosa; May, Jesse; Mayo, John; McDonnell, Mike; McLellan, Susan; McQuarrie, Carolyn; Nood, Julia; Mead, Therese; Meeuwisse, Cory; Meloy, Patrick; Menzies, Perry; Messman, Anne; Miazga, Stephen; Mills, Logan; Milne, Ken; Mix, Allan; Montag, Steve; Moore, Brendon; Morgenstern, Justin; Mott, Sarah; Mukherj, P.; Mulla, Ali; Nandalal, Sheena; Nikel, Taylor; Nugent, Sean; Oakland, Morgan; Oberholzer, Werner; Otugo, Onyeka; Oyedokun, Taofiq Segun; Paddock, Mike; Pardhan, Alim; Patel, Kinjal; Paterson, Quinten; Patocka, Catherine; Patterson, Christine; Pearlman, James; Pelletier-Bui, Alexis; Phan, Marc; Poonja, Zafrina; Powell, Aubrey; Premkumar, Kamini; Prosen, Gregor; Puri, Vishal; Quaife, Tanis; Raffel, Ryan; Raja, Ali; Ramunno, Randi; Rang, Louise; Rannazzisi, Suzanne; Regan, Shauna; Ridderikhof, Milan; Rogers, Vanessa; Roh, Christine; Rosenberg, Keith; Roure, Marina; Rudinsky, Sherri; Rudner, Joshua; Saleh, Adeeb; Sanderson, Will; Scheirer, Owen; Schofield, Paul; Schunk, Paul; Schwarz, Evan; Shahrabadi, Parisa; Shappell, Eric; Sheffield, Julia; Sherbino, Jonathan; Singh, Manpreet; Singson, Hector C.; Slessor, Dave; Smith, Sam; Sneath, Paula; Sobehart, Robert; Spearing, Kerry; Stempien, James; Sternard, Britni; Stratton, Tara; Stuart, Katherine; Stuntz, Bob; Susalla, Michael; Sweeney, Colleen; Swisher, Loice; Swoboda, Henry; Syed, Shahbaz; Taira, Taku; Tambe, Nikhil; Tang, Richard; Targonsky, Elisha; Taylor, Rachel; Taylor, Alan; Taylor, Todd; Ting, Paxton; Tiwald, Gerhard; Tran, Kelvin; Tran, Evelyn; Trickovic, Jason; Trinquero, Paul; Trueger, Seth; Tyagi, Aaron; Umana, Manrique; Vallance, Patrick; Van den Berg, Patricia; Vargas, Luis; Verbeek, Rene; Viggers, Sandra; Vlodaver, Zlata; Wagner, Matthew; Walji, Noorin; Walter, Joe; Wan, Miranda; Wang, Rachel; Wanner, Gregory; Warawa, Wyatt; Ward, Mike; Weekes, Jennifer; Weersink, Kristen; Weessies, Cara; Whalen-Browne, Anna; Whiteside, Brian; Willis, Matthew; Wilmer, Jonas; Wong, Nelson; Woodcroft, Mark; Woods, Rob; Yau, Lawrence; Yee, Jessica; Yeh, Calvin; Huang, Simon York Ming; Yurkiw, Katherine; Zaver, Fareen; Zozula, Alexander
Source
Annals of Emergency Medicine. Sep 01, 2017 70(3):394-401
Subject
Language
English
ISSN
0196-0644
Abstract
STUDY OBJECTIVE:: Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs. METHODS:: We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearsonʼs correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality. RESULTS:: One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students (r=0.92) and residents (r=0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8). CONCLUSION:: The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.