Applying large language models for automated essay scoring for non-native Japanese

This study investigates the use of large language models (LLMs) for automated essay scoring (AES) of non-native Japanese writing. It compares the performance of GPT-4, BERT, a Japanese local LLM (Open-Calm large model), and two conventional machine learning methods (Jess and JWriter) using a dataset of 1400 essays from learners with diverse first languages. GPT-4 significantly outperforms other models in annotation accuracy and proficiency level prediction. The study also highlights the importance of prompt engineering in achieving reliable LLM-based AES.

Wenchao Li, Haitao Liu