Improving the Science of Annotation for Natural Language Processing: The Use of the Single-Case Study for Piloting Annotation Projects

Journal of Data Science

article

Kylie Anglin¹  Arielle Boguslav²  Todd Hall² 

[1] Department of Educational Psychology, University of Connecticut;Department of Education Leadership Foundations and Policy, University of Virginia

关键词: annotation;    coding;    single-case study;    supervised machine learning;    text classification;   

DOI  :  10.6339/22-JDS1054

学科分类：土木及结构工程学

来源: JDS

【 摘 要 】

Researchers need guidance on how to obtain maximum efficiency and accuracy when annotating training data for text classification applications. Further, given wide variability in the kinds of annotations researchers need to obtain, they would benefit from the ability to conduct low-cost experiments during the design phase of annotation projects. To this end, our study proposes the single-case study design as a feasible and causally-valid experimental design for determining the best procedures for a given annotation task. The key strength of the design is its ability to generate causal evidence at the individual level, identifying the impact of competing annotation techniques and interfaces for the specific annotator(s) included in an annotation project. In this paper, we demonstrate the application of the single-case study in an applied experiment and argue that future researchers should incorporate the design into the pilot stage of annotation projects so that, over time, a causally-valid body of knowledge regarding the best annotation techniques is built.

【 授权许可】

CC BY