TY - JOUR
T1 - In Validations We Trust? The Impact of Imperfect Human Annotations as a Gold Standard on the Quality of Validation of Automated Content Analysis
AU - Song, Hyunjin
AU - Tolochko, Petro
AU - Eberl, Jakob Moritz
AU - Eisele, Olga
AU - Greussing, Esther
AU - Heidenreich, Tobias
AU - Lind, Fabienne
AU - Galyga, Sebastian
AU - Boomgaarden, Hajo G.
N1 - Funding Information:
Olga Eisele is supported by the Austrian Science Fund under the Hertha-Firnberg-Program [Grant no: T-989].
Publisher Copyright:
© 2020, © 2020 The Author(s). Published with license by Taylor & Francis Group, LLC.
PY - 2020
Y1 - 2020
N2 - Political communication has become one of the central arenas of innovation in the application of automated analysis approaches to ever-growing quantities of digitized texts. However, although researchers routinely and conveniently resort to certain forms of human coding to validate the results derived from automated procedures, in practice the actual “quality assurance” of such a “gold standard” often goes unchecked. Contemporary practices of validation via manual annotations are far from being acknowledged as best practices in the literature, and the reporting and interpretation of validation procedures differ greatly. We systematically assess the connection between the quality of human judgment in manual annotations and the relative performance evaluations of automated procedures against true standards by relying on large-scale Monte Carlo simulations. The results from the simulations confirm that there is a substantially greater risk of a researcher reaching an incorrect conclusion regarding the performance of automated procedures when the quality of manual annotations used for validation is not properly ensured. Our contribution should therefore be regarded as a call for the systematic application of high-quality manual validation materials in any political communication study, drawing on automated text analysis procedures.
AB - Political communication has become one of the central arenas of innovation in the application of automated analysis approaches to ever-growing quantities of digitized texts. However, although researchers routinely and conveniently resort to certain forms of human coding to validate the results derived from automated procedures, in practice the actual “quality assurance” of such a “gold standard” often goes unchecked. Contemporary practices of validation via manual annotations are far from being acknowledged as best practices in the literature, and the reporting and interpretation of validation procedures differ greatly. We systematically assess the connection between the quality of human judgment in manual annotations and the relative performance evaluations of automated procedures against true standards by relying on large-scale Monte Carlo simulations. The results from the simulations confirm that there is a substantially greater risk of a researcher reaching an incorrect conclusion regarding the performance of automated procedures when the quality of manual annotations used for validation is not properly ensured. Our contribution should therefore be regarded as a call for the systematic application of high-quality manual validation materials in any political communication study, drawing on automated text analysis procedures.
UR - http://www.scopus.com/inward/record.url?scp=85081350778&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85081350778&partnerID=8YFLogxK
U2 - 10.1080/10584609.2020.1723752
DO - 10.1080/10584609.2020.1723752
M3 - Article
AN - SCOPUS:85081350778
JO - Political Communication
JF - Political Communication
SN - 1058-4609
ER -