In Validations We Trust? The Impact of Imperfect Human Annotations as a Gold Standard on the Quality of Validation of Automated Content Analysis

Hyunjin Song, Petro Tolochko, Jakob Moritz Eberl, Olga Eisele, Esther Greussing, Tobias Heidenreich, Fabienne Lind, Sebastian Galyga, Hajo G. Boomgaarden

Research output: Contribution to journalArticlepeer-review

8 Citations (Scopus)

Abstract

Political communication has become one of the central arenas of innovation in the application of automated analysis approaches to ever-growing quantities of digitized texts. However, although researchers routinely and conveniently resort to certain forms of human coding to validate the results derived from automated procedures, in practice the actual “quality assurance” of such a “gold standard” often goes unchecked. Contemporary practices of validation via manual annotations are far from being acknowledged as best practices in the literature, and the reporting and interpretation of validation procedures differ greatly. We systematically assess the connection between the quality of human judgment in manual annotations and the relative performance evaluations of automated procedures against true standards by relying on large-scale Monte Carlo simulations. The results from the simulations confirm that there is a substantially greater risk of a researcher reaching an incorrect conclusion regarding the performance of automated procedures when the quality of manual annotations used for validation is not properly ensured. Our contribution should therefore be regarded as a call for the systematic application of high-quality manual validation materials in any political communication study, drawing on automated text analysis procedures.

Original languageEnglish
JournalPolitical Communication
DOIs
Publication statusAccepted/In press - 2020

Bibliographical note

Funding Information:
Olga Eisele is supported by the Austrian Science Fund under the Hertha-Firnberg-Program [Grant no: T-989].

Publisher Copyright:
© 2020, © 2020 The Author(s). Published with license by Taylor & Francis Group, LLC.

All Science Journal Classification (ASJC) codes

  • Communication
  • Sociology and Political Science

Fingerprint

Dive into the research topics of 'In Validations We Trust? The Impact of Imperfect Human Annotations as a Gold Standard on the Quality of Validation of Automated Content Analysis'. Together they form a unique fingerprint.

Cite this