Asian Association for Language Assessment

While test developers have long embraced technology in their development, delivery and scoring processes, the recent popularity of artificial intelligence, particularly in the form of large language models (LLMs), has brought with it exciting opportunities. However, the new technologies have also highlighted a number of significant challenges, some of which pre-dated the current technology cycle while others are genuinely new.

In this talk, I present an overview of a major initiative undertaken by the British Council to create interactive language speaking tasks to a set specification using LLM technology. The first of these tasks were made available to learners for formative purposes in April 2024. While the talk will present insights into the approach taken, the primary focus will be on ethical and accessibility issues that emerged during the process.

In dealing with the ethical side of the application of AI in assessment, we realised from the outset, that we did not have sufficient expertise. With this in mind, we partnered with the Knowledge Lab at University College London who worked with us to develop a practical (if complex) tool which was used to perform an ethics audit of all aspects of our approach and our aims. This proved invaluable in helping us devise an ethics-by-design approach. Within this approach, we also focused on ensuring the accessibility of our final product, taking the British Council’s Justice, Equality, Diversity and Inclusion (JEDI) policies and practices into account. Uniquely, we have two formal written policies, one is person-centred (e.g. special accommodations) and the other is test-centred (e.g. representation in test content).

The talk concludes by stressing the importance of the approach taken to the application of AI in language assessment, while also highlighting the dangers of ignoring or relegating the importance of ethics and accessibility within such test development and delivery contexts.