A Study on Rater Reliability Under Holistic and Analytic Scoring of CEPT Writing by Using Generalizability Theory and Many-facet Rasch Model
- DOI
- 10.2991/emle-18.2018.201How to use a DOI?
- Keywords
- placement test; writing scoring reliability; holistic scoring; analytic scoring; generalizability theory; many-facet Rasch model
- Abstract
To explore the scoring reliability of College English Placement Test (CEPT) writing, generalizability theory (GT) and many-facet Rasch model (MFRM) were applied to analyze 15 raters’ holistic and analytic ratings of 300 writing samples. The results were as follows: (1) Raters’ scoring method had significant impact on their rating of CEPT writing; under either method, one rater was enough to ensure that the generalizability coefficient was 0.8 or above. (2) Whichever method was adopted, raters differed significantly from each other in severity, but they had sound intra-rater consistency; raters were most biased towards task, then grammar, mechanics, vocabulary, and were least biased towards structure; if the dimensions at the discourse level were scored severely, then those at the linguistic level were scored leniently, and vice versa; raters tended to be severely biased towards the low, the intermediate level groups, and the lowest proficiency examinee, while be leniently biased towards the high level group, and the highest proficiency examinee. GT and MFRM proved from macro- and micro-levels respectively that the CEPT writing scoring enjoyed high reliability.
- Copyright
- © 2018, the Authors. Published by Atlantis Press.
- Open Access
- This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
Cite this article
TY - CONF AU - Lin Chun AU - Xiao Yunnan PY - 2018/12 DA - 2018/12 TI - A Study on Rater Reliability Under Holistic and Analytic Scoring of CEPT Writing by Using Generalizability Theory and Many-facet Rasch Model BT - Proceedings of the 4th International Conference on Economics, Management, Law and Education (EMLE 2018) PB - Atlantis Press SP - 1013 EP - 1019 SN - 2352-5428 UR - https://doi.org/10.2991/emle-18.2018.201 DO - 10.2991/emle-18.2018.201 ID - Chun2018/12 ER -