Asian Association for Language Assessment

Language assessment literacy papers and workshops have focused on how  to make participants understand concepts written as standards. One  popular set of standards are the ones written by psychological and  educational experts and best practices in the field (APA, AERA, NCME, Standards for educational and psychological testing,  1999, and 2014). The standards include validity, reliability,  generalizability, fairness, norm and criterion-referenced assessment,  etc. These standards have provided assessment institutions guidance for  their own internal evaluations and research agendas. But, students and  young professionals are expected to understand these concepts without  guiding principles. More recently, the argument-based approach based on  Toulmin's model (example, Bachman and Palmer, 2010) has offered a  systematic approach to evaluation with an examination of an assessment  institution's claims and warrants, and the backing for warrants. But,  neither does this approach offer guiding principles. Thus, these two  top-down approaches are unlikely to be able to help participants  understand key assessment knowledge.

​

In  order to remedy this situation, I propose a series of reflections on  hypothetical scenarios that can lead us to critical guiding principles.  These guiding principles can further take us to the development of  claims and evidence for support. In this approach, first, a series of  scenarios called "The Trolley Problem" (Foot, 1967) from moral  philosophy will provide an introduction as to how to evaluate these  scenarios – whether to morally justify actions based on the principle of  outcomes/consequentialism or to use the principle of duty/obligation.  Second, six scenarios from language assessment on defective tasks,  biased tasks, scoring problems, selecting an assessment, differential  pricing, and decision-making will be analyzed by applying outcomes-based  or duty-based thinking. As these scenarios mirror the common assessment  development to assessment decision-making process, applying principles  to these scenarios will be quite transparent. Third, participants will  formalize their understanding by checking a list of concepts/standards  that include different aspects of validity, reliability, and fairness.  Therefore, this approach will help participants understand key  assessment knowledge through a bottom-up approach.