In his introduction, Weir states, "The core of this book is concerned with exploring a framework for establishing the validity of the interpretation of scores on tests produced by Exam Boards or by teachers for use in their classrooms." The "evidence-based approach" in the subtitle alludes to the view that the process of validating the use of scores on any given test is much like a courtroom trial − requiring evidence to support arguments for or against a favorable verdict that the test was "fair". In first of the book's four parts, Weir introduces five ways to collect the evidence needed to make a convincing case; the first two are collected a priori, or in the design phase of a test, and consist of defining what abilities the test is supposed to measure as well as how the sample of tasks in the test represents the abilities in 'the real world' (outside the test itself) that test users are looking for. The remaining three types of evidence are empirical, collected a posteriori as statistical procedures to estimate or enhance the reliability of the test scores, studies of how well test scores correlate with external criteria such as other tests purported to measure the same abilities or actual performance in the real world, and lastly, the study of the backwash or social consequences of test use for the stakeholders: teachers, students, parents, administrators, and "the marketplace". Part two begins a detailed survey of frameworks for tests of reading, listening, speaking and writing. Each framework is presented as a flow-chart of boxes which detail the a priori considerations − test taker characteristics, test characteristics, theories of internal processes and resources − to the a posteriori considerations for investigating scoring characteristics, criterion-related evidence of score value and the impact of score interpretation. Six chapters present examples from actual research to illustrate how evidence of validity was obtained "in action". Part three could serve as a syllabus for a practicum on test validation methodology. It contains pointers for sound research procedures, checklists and questionnaires to help researchers collect each of the five types of evidence for valid test score use. This part alone would justify buying the book, as it can certainly encourage teachers to not only have more confidence in their own assessment practices, but would probably result in some pretty interesting presentations or publications. Part four of the book is entitled "Further resources in language testing", and is actually an up-to-date and comprehensive list of textbooks, journals, professional organizations, professional conferences, e-mail lists, bulletin boards and websites, databases and statistical packages − a list probably equal to a few years of word-ofmouth searches for answers to the questions that plague anyone who gets involved with testing and assessment. In conclusion, this book is highly recommended. It is the most readable and comprehensive treatment of validation that I have ever come across. It is as free of technical jargon as can be; important points are presented clearly through Weir's choice to enclose poignant quotes and concepts in boxes within the text which saves on magic markers or underlining, in addition to the other features of this book that have already been reported. Its 284 pages contain not a single mathematical equation or algorithm, so it's not a cookbook for number crunching, but it'll tell you where to go for that if you need it − plus much, much more.

/pdf/language-testing-and-validation-an-evidence-based-approach-3nrcal24dk.pdf

Language Testing and Validation: An Evidence-Based Approach

Language testing and validation

Key to symbols Preface Acknowledgements 1. General Characteristics 2. Interactional structure 3. Interactional strategies 4. Conversation 5. Discourse and grammar Answers to the exercises Glossary References Index

An introduction to spoken interaction

Language testing and assessment have moved center stage in recent years, whether for educational, employment, or sociopolitical reasons More and more people are involved in developing tests and using test score outcomes, though often without a background or training in assessment to equip them adequately for this role Simultaneously, increasing professionalization of the field has led to the generation of standards, ethical codes, and guidelines for good testing practice Although these can help make assessment practices more transparent and accessible to a wider constituency, they also risk promoting a view of language testing as highly technical and specialized–best left to experts These trends have implications for both policy and practice This article reviews efforts to promote understanding of assessment within the field of applied linguistics and within education and society more broadly The role of professional associations, academic institutions, and commercial organizations in developing assessment literacy is considered, as well as the contribution of published material and other types of training resources This article reflects on how the international language testing community can encourage the sharing of the core knowledge, skills, and understanding that underpin good quality assessment as widely and accessibly for the benefit of all

/pdf/developing-assessment-literacy-4xw9625ucx.pdf

Developing assessment literacy

In Part 1 of this two-part review article (Alderson & Banerjee, 2001), we first addressed issues of washback, ethics, politics and standards. After a discussion of trends in testing on a national level and in testing for specific purposes, we surveyed developments in computer-based testing and then finally examined self-assessment, alternative assessment and the assessment of young learners.In this second part, we begin by discussing recent theories of construct validity and the theories of language use that help define the constructs that we wish to measure through language tests. The main sections of the second part concentrate on summarising recent research into the constructs themselves, in turn addressing reading, listening, grammatical and lexical abilities, speaking and writing. Finally we discuss a number of outstanding issues in the field.

https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S0261444800014464

Language testing and assessment (Part 2)

This paper explores the effect on pair-task performance of test-takers’ familiarity with their partner. O’Sullivan (2000a), like McNamara (1996), sees test performance as being affected by a number of factors, related to the test-taker, the interlocutor and the task. Evidence from the psychology literature, and anecdotal evidence from language learners and teachers, suggests that familiarity with one’s partner in an interactive task might positively affect performance, although in the only published study to date, Porter (1991a) found no evidence to support this hypothesis.A group of 32 Japanese learners performed a series of three tasks (personal information exchange, narrative, and decision making), once with a friend, and again with a person who was not known to them. All performances were video-recorded, awarded scores by trained raters and transcribed for analysis. Results suggest evidence of an `acquaintanceship’ effect, with subjects achieving higher scores when working with a friend. However, anal...

Learner acquaintanceship and oral proficiency test pair-task performance:

Test-task validation has been an important strand in recent revision projects for University of Cambridge Local Examinations Syndicate (UCLES) examinations. This article addresses the relatively neglected area of validating the match between intended and actual test-taker language with respect to a blueprint of language functions representing the construct of spoken language ability. An observation checklist designed for both a priori and a posteriori analysis of speaking task output has been developed. This checklist enables language samples elicited by the task to be scanned for these functions in real time, without resorting to the laborious and somewhat limited analysis of transcripts. The process and results of its development, implications and further applications are discussed.

Using observation checklists to validate speaking-test tasks

Exploring Gender and Oral Proficiency Interview Performance.

Performance in tests of spoken language can be influenced by a wide range of features of both task and participants. This article hypothesizes that there may be effects on performance attributable to an interaction of variables such as the task topic, the gender of the person presenting the topic and the gender of the candidate. In contrast to previous studies, which have examined speaking tests involving face-to-face interaction, this study considered the issue in a tape-mediated test delivered in a language laboratory, with no inter-locutor present, but where stimulus material is presented by one or more speakers, one of whom acts as ‘audience’ for the candidate’s speech. The test was taken by 894 students graduating from Hong Kong universities. In an advice-giving task, the last of a series involving different situations and audiences, topics considered stereotypically ‘male’-oriented or ‘female’-oriented were presented with systematic changes in the roles taken by the male and female speakers. A multi...

Barry O'Sullivan

Papers

Learner acquaintanceship and oral proficiency test pair-task performance:

Using observation checklists to validate speaking-test tasks

Exploring Gender and Oral Proficiency Interview Performance.

The Effect of Test-Taker Gender, Audience and Topic on Task Performance in Tape-Mediated Assessment of Speaking.

The effect of test-taker sex, audience and topic on task performance in tape-mediated assessment of speaking