Asian Association for Language Assessment

This paper introduces current topics and research in Artificial Intelligence (AI) techniques for the scoring of speaking and writing in English language assessments. The paper is organized around several main themes which are central to the use of AI. 

First, the importance of “training” data required to develop AI scoring engines is discussed. There is a growing awareness of the importance of representative training data, and how bias in training data results in bias outcomes. We will investigate examples where sparse data or data unrepresentative of the target test audience may lead to potentially damaging results, for example, uneven performance in the accuracy of speech recognition across different populations. 

Next, the features that comprise automated scoring models are analyzed, for both writing and speaking. We will review the commonly-used features (or variables) in scoring models and compare them with typical descriptors in rating criteria to evaluate their fit with the intended construct. Also examined are the differences between “surface features” such as punctuation or vocabulary frequency which are readily understandable - and often gameable - versus harder-to-gauge techniques from the field of natural language processing. The visible, verifiable approach of feature analysis and modeling (“white box”) is contrasted with deep learning approaches where the inner workings of the algorithms are difficult for humans to trace (so-called “black box”).

Last, validity is discussed with reference to the kind of additional analyses that we should expect to see in AI-scored assessments. While the gathering of validity evidence is an ongoing process, there are certain crucial data which allow stakeholders to properly evaluate a test that should be reported from the outset of test publication, particularly with regard to the performance of machine scoring and interpretation of AI-based scores.

Throughout this presentation, the benefits and limitations of certain approaches to AI scoring are described, with a view to educating the audience so that they are better able to understand and evaluate AI-scored assessments.