All Journals

Language Teaching Research Quarterly

European Journal of Studies in Management and Business

International Journal of Behavior Studies in Organizations

Studies in Educational Management

New Challenges in Accounting and Finance

Language Testing in Focus: An International Journal

Marketing and Branding Research

Individual Differences in Language Education: An International Journal 

Feedback Research in Second Language 

Management Issues in Healthcare System

Language Teacher Education Research

Artificial Intelligence Knowledge Development

<p style="text-align:justify;">This paper extends the use of Generalizability Theory to the measurement of extemporaneous L2 speech through the lens of speech perception. Using six datasets of previous studies, it reports on G studies–a method of breaking down measurement variance–and D studies–a predictive study of the impact on reliability when modifying the number of raters, items, or other facets that assist the field in adopting measurement designs that include comprehensibility, accentedness, and intelligibility. When data from a single audio sample per learner were subjected to D-studies, we find that both semantic differential and rubric scales for comprehensibility were reliable at the .90 level with about 15 trained raters or 50 untrained crowdsourced raters. In order to offer generalizable and dependable evaluations, empirically informed recommendations are given, including considerations for the number of speech samples rated, or the granularity of the scales for various assessment and research purposes.</p>

ltrq.2023.37.12.pdf

Original Research 

ltrq.2023.37.12.xml

How Many Raters Can Be Enough: G Theory Applied to Assessment and Measurement of L2 Speech Perception

Guest-editors: 
Christine Coombe, PhD, Higher Colleges of Technology, Dubai, UAE
Tony Clark, PhD, Cambridge University Press & Assessment, UK
Hassan Mohebbi, PhD, Independent Researcher, Turkey


Special Issue: In Honour of James Dean Brown’s Five-Decade Contribution to Language Testing