Conceptualizing and operationalizing the construct of critical thinking in speaking assessments of English as a foreign language

This joint PhD project is based at the Shanghai Jiao Tong University with a minimum 12 month stay at The University of Melbourne.

Project description
This study aims to conceptualize critical thinking ability and operationalize the construct in L2 speaking assessments. Following the literature review, the researchers will survey current practices of speaking assessments to identify the contextual features that would best elicit critical thinking performances. Tasks and rating scales will be designed and validated to gain    a better understanding of the critical thinking ability and its operationalization in speaking assessments. It is hoped that the study will broaden the construct of speaking assessments by incorporating the critical thinking ability and throw light on the evaluation of test takers’ critical thinking performances. This study sets out to examine how the construct of critical thinking can be operationalized in language assessments targeted at learners of English as a second or foreign language.

Key questions and hypotheses

Drawing on Paul and Elder’s (2001) three-dimension model, the construct of critical thinking is defined as a conscious application of a framed way of thinking which aims to improve the quality of thinking and achieve intellectual standards of excellence. The conceptualization has incorporated the three key dimensions of critical thinking: cognitive skills, dispositions, and standards for evaluating the quality of critical thinking. This working definition will guide the design of assessment tasks and the development of rating scales of the critical thinking ability.

Three research questions (RQ) will be answered in the study.

  1. To what extent is the critical thinking ability assessed in English speaking assessments currently in use? The hypothesis is that the current design of speaking assessment tasks and the rating scales have failed to tap into candidates’ ability to think critically.
  2. How can the construct of critical thinking be best represented in speaking assessment tasks? It is expected that with appropriate contextual features and task design, critical thinking performances can be elicited in speaking assessments for an accurate and reliable evaluation.
  3. What are the criteria for evaluating critical thinking performances and how can rating scales be developed and validated? It is expected that evaluation criteria and rating scales can be generated and validated through an empirical investigation of learners’ performances and raters’ perceptions of the key features of their performances.

Scope of work to be undertaken at each institution and strengths of each institution’s contribution to the project

  • To answer RQ1, a test review is to be conducted by the SJTU team to see whether and to what extent the critical thinking ability has been incorporated into the construct of the speaking tasks of international or local language tests, including for example, TOEFL iBT in the US; IELTS, Aptis, PTE-Academic, and Linguaskill in the UK; OET in Australia; and CET-SET in China.
  • To address RQ2, the SJTU team will be responsible for identifying and specifying the contextual features of assessment tasks on critical thinking. Assessment tasks will be developed and piloted among the participants in China. The SJTU team has been running the College English Test (CET), which has over 20 million test takers a year and is the largest test of English as a foreign language in the world, for over three decades, and is therefore very experienced in assessment design and development. In this study, critical thinking tasks will be incorporated into the CET Spoken English Test (CET- SET). Pilot studies will be conducted by the SJTU team.   In the main study of validating the assessment tasks, data will be collected from two groups of participants: pre-sessional students at the University of Melbourne (n=100) and the CET-SET test takers in China (n=200).

The variety of participants’ characteristics, including their L1 background, level of English proficiency, motivation of learning English and so on, will improve the robustness of the task design.

  • RQ3 looks into the evaluation of the critical thinking ability. For performance-based assessments, a well- designed rating scale enables maximal construct representation. The UoM team will be consulted at this stage. To validate the rating scales in a principled way, we will use an argument-based framework proposed by Knoch and Chapelle (2018). Validity evidence will be collected to argue for the evaluation, generalization, explanation, and extrapolation inferences in the framework. Follow-up interviews and questionnaire surveys will also be conducted among test takers and raters at both SJTU and UoM for their perceptions of the construct being tapped into by the rater scales.

The project will be complemented by the project Expanding the construct of second language assessments and the collaboration will ensure a successful completion of the project.

Supervision team:

Professor Yan Jin (Shanghai Jiao Tong University)

Associate Professor Ute Knoch (The University of Melbourne)