Asian Association for Language Assessment

In  second language (L2) pronunciation assessment, the consequences of an  intuitive/experiential approach to rating scale development have led to  shortcomings in the quality of the pronunciation descriptors used in  current scales (Isaacs, 2013). For example, the main CEFR scales, which  were compiled from assorted intuitively-derived descriptors, exclude  reference to pronunciation, partially reflecting the inadequacy of those  descriptors. Speaking scales that include a pronunciation component are  also problematic. Some haphazardly describe behavioural indicators  across levels (e.g., ACTFL), whereas others are so general that the  specific linguistic features that constitute level distinctions are  often unclear (e.g., IELTS). Still other scales imply or directly equate  increased intelligibility (i.e., understandability of L2 speech) with  the reduced presence of a foreign accent (e.g., CEFR phonological  control scale). However, this practice contradicts strong research  evidence that perfectly intelligible speech does not preclude the  presence of a noticeable L2 accent, whereas a heavy accent is a hallmark  of unintelligible speech (Derwing & Munro, 2009). Developing an  evidential basis for operationalizing pronunciation features in rating  scales is essential for generating more valid assessments.

This  paper will report on a research program on using raters’ perceptions  and judgments of L2 speech to better understand the linguistic  properties that underlie speech that is easily understandable (often  termed ‘intelligible’ or comprehensible’ in rating scales). After  presenting findings from an initial scale development study  involving  ‘lay’ raters’ judgments of L2 French learners’  comprehensibility and a follow-up study on speaker first language  effects (Crowther et. al, 2014), the paper will turn to focus group and  rating data from eight accredited IELTS examiners on their impressions  of the IELTS Pronunciation scale. Implications for pronunciation scale  development and validation, including challenges in teacher-raters’  pronunciation literacy, will be discussed.