Asian Association for Language Assessment

Evaluations of language assessments often use the concept of fairness. The Standards for Educational and Psychological Testing (AERA, APA, NCME, 1999) with a section titled “Fairness in Testing” and Codes of Ethics and Practice in assessment agencies such as ILTA, ETS, and ALTE include the concept of fairness. Recent publications have included fairness:  situated ethics (Kunnan & Davidson, 2004) and how to investigate fairness (Xi, 2010). The term justice is not as well known in the assessment literature although the idea of justice has been discussed in writings from Plato to Rawls and Sen. The term includes “distributive justice” which refers to institutions providing benefits that are distributed to a society in a just manner. In language assessment, Kunnan has tied the two concepts together (Kunnan, 2004, 2008, 2010, 2014) and McNamara and Roever (2010) have offered separation and clarity.

Based on work by Rawls and Sen, this talk will present principles and sub-principles of fairness and justice for evaluation of language assessments. It will apply the idea of fairness as relating to persons - how assessments ought to be fair to test takers, and the idea of justice as relating to institutions - how institutions ought to be just to test takers.

Principle 1: The Principle of Fairness: An assessment ought to be fair to all test takers.

• Sub-principle 1: An assessment ought to provide adequate opportunity to learn the knowledge, abilities or skills to be assessed for all test takers.   

• Sub- principle 2: An assessment ought to be consistent and meaningful in terms of its test-score interpretation for all test takers.

• Sub- principle 3: An assessment ought to be free of bias against all test takers, in particular by assessing construct-irrelevant matters.

• Sub- principle 4: An assessment ought to use appropriate access, administration, and standard setting procedures so that decision-making is equitable for all test takers.

Principle 2: The Principle of Justice: An assessment institution ought to be just.

• Sub-principle 1: An assessment institution ought to bring benefits to society by making a positive social impact.

• Sub-principle 2: An assessment institution ought to advance justice through public reasoning of its assessment.

A discussion of these principles, the sub-principles and the warrants that go with them (in the Toulmin (1953) argumentation model of grounds, warrants, backing, rebuttals, etc.; Kane, 2010, Bachman (2005) is presented. Finally, sample empirical studies are presented and mapped onto the Toulmin model so that evaluations can be made regarding claims of fairness and justice of particular language assessments.