Asian Association for Language Assessment

From the time technology being seen as a source of construct irrelevance variance (Taylor et al., 1998) to generative AI tools being advocated for integration into the construct (Voss et al., 2023; Xi, 2023), conceptions of language tests may have shifted beyond imagination within just around 25 years due to accelerated breakthroughs in technology. While serious debates regarding the nature of language constructs have been happening that conjecture what our next generation of language tests should look like, our large-scale testing practices have remained almost the same as decades ago. As language testing professionals, how do we work together to push the boundaries of language tests?

Language tests are expected to mimic real-world communication as much as feasible. Nowadays the widespread use of assistive and even generative AI tools by language users in real life has motivated new inquiries into the nature of language ability and started to push language testers to revisit the nature of language competence and how our language testing practices can be transformed.  

In this talk, I will first clarify the use of technology as a medium or aid of communication in real life vs. a tool to enable test delivery, as for a long time they may have been conflated. This distinction is important as we ponder over the exact role technology plays in the construct definition of language tests.  

Expanding on Xi (2024), I then delineate four approaches to addressing the role of technology in defining language test constructs, Outright Rejection, Forced Acceptance, Cautious Acceptance, and Progressive Embracing. The first approach treats technology as a potential source of construct irrelevance. A good example is using a paper and pencil format to test writing skills, driven by the conception that writing conventions such as spelling and handwriting skills are a core part of writing ability and keyboarding skills are not pertinent   to the measurement of writing. The second approach recognizes the role of technology to a limited extent by accepting basic computer literacy skills (e.g., reading on the computer screen and keyboarding skills) implicitly as part of the constructs, mostly motivated by the need to use computer-based delivery to scale up testing.  This approach is most commonly adopted in large-scale testing, although test designers would still be concerned about the potential impact of technology skills on test performance. The Cautious Acceptance approach, which may accept controlled use of assistive tools by test takers such as spelling and grammar checkers and online dictionaries, has not made its way into large-scale testing yet. The final approach takes the most progressive stance and defines language constructs as communication skills fully integrated with computer and digital information literary skills, such as the ability to effectively use the full set of editing tools and even generative AI tools to accomplish a communication task, and to use digital technologies to search and identify needed information and evaluate, organize, and synthesize it to fulfill a task. This final approach, although having been advocated by researchers, seems to be far-fetched for real-world applications currently.

What are some of the barriers to the conceptualization and operationalization of the last two approaches? I believe some conceptual challenges remain to be resolved towards shifting our long-established thinking about language constructs. The conventions of language, such as writing conventions or even handwriting skills, remain a core component of writing curriculum around the world and what we test in writing, for example. The use of any assistive tools may “contaminate” or “complicate” the assessment of language skills, a view held by many mainstream language testers. The use of generative tools, such as ChatGPT, has often been seen as cheating tools and banned in many classroom assignments and assessments. Differential exposure to technology and technological tools and opportunities to use and practice with technological tools would also introduce potential fairness issues, a major hurdle to innovation in large-scale testing.

Providing learners and test takers access to these tools would be a leap of faith for many language educators and testers and make them uncomfortable as the speed at which AI technology advances has far out-paced evolutions of language constructs and testing practices. Language classroom teachers have been more adaptive, taking the plunge in using these AI tools in classroom assessment. Contrary to the common practice of using innovations in large-scale testing as a catalyst for language education reforms, designers of formative assessment will perhaps lead the way in this digital era, preparing the ground for possible future changes in the practice of large-scale language tests.