PNRE: Proactive Learning for Neural Relation Extraction with Multiple Annotators

DOI

10.2991/978-94-6463-046-6_42How to use a DOI?

Keywords

Proactive Learning; Relation Extraction; Cost-Effectiveness

Abstract

Relation extraction is one of the essential tasks of information extraction, and it is also a fundamental part of knowledge graph construction. Many works have been proposed for supervised relation extraction, which commonly requires a massive amount of human-annotated data with both time and cost. To reduce the labeling time and cost, active learning has been proposed with the assumption of a single perfect annotator that always furnishes the correct label. However, more generally, the annotator will provide incorrect labels according to their labeling capabilities, and different labeling capabilities correspond to distinct costs. To unleash the power of annotators with diverse expertise level and unlabeled data for better model performance with the lowest cost, we develop PNRE, a novel proactive learning based framework for neural relation extraction that actively select the most suitable sample-annotator pairs to construct high-quality relation extraction corpus. Specifically, PNRE utilizes (1) Expert Performance Estimation module to precompute each annotator’s performance considering class prediction probability; (2) Sample Selection module to select the most informative and representative sample based on a hybrid query strategy; (3) Sample Allocation module to allocate appropriate sample to each annotator under the condition of annotation utility maximization. The framework is evaluated on three corpora and is shown to achieve promising results with a significant reduction in labeling costs.

Copyright

© 2023 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Rui Qiu
AU  - Wen Ji
AU  - Yundan Liang
AU  - Haini Qu
AU  - Jingce Xu
PY  - 2022
DA  - 2022/12/17
TI  - PNRE: Proactive Learning for Neural Relation Extraction with Multiple Annotators
BT  - Proceedings of the 2022 2nd International Conference on Computer Technology and Media Convergence Design (CTMCD 2022)
PB  - Atlantis Press
SP  - 352
EP  - 364
SN  - 2352-538X
UR  - https://doi.org/10.2991/978-94-6463-046-6_42
DO  - 10.2991/978-94-6463-046-6_42
ID  - Qiu2022
ER  -