How Pisa’s Standardised Questions Break Down in the Hands of 15-Year-Old Test Takers.
Over the years, PISA tests have been researched from a number of perspectives including their psychometric validity and reliability, their policy impacts, and their reception among educators and policymakers. However, there is still a lack of investigation into the factors that shape students’ responses on the assessment. The few scholars that have forayed into this area (for example, Eklöf & Hopfenbeck, 2019; Hopfenbeck & Maul, 2011; Serder & Jakobsson, 2015). These researchers argue that, as it is the students’ performance which provides a picture of a country’s education system, the ways in which students understand and respond to the assessment is an important aspect to investigate. The research described here by Harsha Chandir contributes to this area as it takes the PISA 2018 Global Competence questionnaire and explores how a selection of 15-year-old students in Victoria, Australia, understand and respond to the questions inscribed in it.
PISA’s standardised measure
What does it mean to be globally competent? Existing literature shows that the notion of global competence is complex and multifaceted and conveys diverse understandings. Additionally, this notion is interwoven with other terms including intercultural competence (Deardorff, 2006; Halse et al., 2015; UNESCO, 2013), international mindedness (International Baccalaureate Organization, 2017), global citizenship (Oxfam, 2015; Oxley & Morris, 2013), intercultural understanding (Australian Curriculum, n.d.), and intercultural capability (Victorian Curriculum Assessment and Authority, n.d.). However, this multiplicity is problematic for standardised comparative assessments, which must go through the processes of settling on specific definitions (and therefore supressing others), and then translating those definitions into indicators and measurement instruments.
The OECD’s assessments can be seen to curtail multiplicity as they marshal different views towards one definition and one interpretation to stabilise the concept. In order to standardise the constructs and render them comparable across diverse contexts, the test-takers first needed to be standardised themselves. Thus, the assessment would need to construct an abstraction of 15-year-olds around the world (Gorur, 2016). The OECD’s 2018 PISA uses the following definition of global competence:
Global competence is the capacity to examine local, global and intercultural issues, to understand and appreciate the perspectives and world views of others, to engage in open, appropriate and effective interactions with people from different cultures, and to act for collective well-being and sustainable development. (OECD, 2018, p. 7)
PISA has come to be trusted as an accurate and valid comparative measurement, and these claims are backed by the range of global experts it gathers to produce its tests, and the field testing and other quality assurance measures it adopts. However, in the process of developing this assessment, the categories and conventions used to measure global competence first need to be developed, in order to make them comparable across diverse countries. As Cardoso & Steiner-Khamsi (2017, p. 401) have stated, phenomena are “not comparable per se, but they are made comparable by using the same measurement.” In making global competence comparable across countries, the phenomenon is “inevitably stripped of [its] context, history, and meaning” (Merry, 2016, p. 1). For example, in developing a single, global instrument to assess literacies across diverse countries, the OECD has been criticised for promoting standardisation over contextually meaningful data that are specific and rich enough to inform policy (Meyer et al., 2014).
In large-scale comparative assessments, including PISA, Dohn argues that “the expectation that there will be no test item that can be found to be biased or ambiguous is probably too severe” (Dohn, 2007, p. 11). This means that an item that shows some level of bias or ambiguity is not, in itself, something scandalous. At the same time, it is important and instructive to explore the nature and the extent of the biases, the assumptions, and the ambiguities presented in an assessment. Such information is important when interpreting the subsequent results of the assessment.
The method of survey encounters provides glimpses into how a selection of 15-year-old students in Victoria, Australia, understand and respond to PISA’s global competence survey, building on methods used in previous studies, specifically by Serder and Jakobsson (2015) and Karabenick et al. (2007). The survey encounter has two main parts: (1) students in groups of three are given a sample of questions from the student background questionnaire and are given the following instructions:
Please read the question out loud, discuss what you think the question is asking of you, which answer would you choose as the right answer for you? Explain why you chose that answer.
A follow up semi-structured interview is then carried out with each participant.
Empirically studying the encounters between test takers and the assessment can provide rich data which cannot be captured in psychometric analyses (Maddox 2015; Serder & Jakobsson 2015). Thus, this method offers a way of “testing the test,” by demonstrating the extent to which the standardised questions are able (or not) to maintain their standardisation in the hands of the 15-year-olds in the study. Moreover, they have the added benefit of highlighting the contextually rich practices shaping different enactments of global competence as the following examples illustrate.
The examples below demonstrate that some of the experiences of 15-year-olds are too complex to be easily answered with a binary yes/no.
Acting for sustainable development?
One aspect of PISA’s definition of a globally competent individual is the extent to which they are able to “act for collective well-being and sustainable development” (OECD, 2018, p. 7). Translating this aspect into the survey, students are asked to report on their engagement with global issues. Test takers are asked if they were involved in a number of activities and had to respond whether the statement described them or not (Yes/No). One of the activities listed was – I reduce the energy I use at home (e.g. by turning the heating down or turning the air conditioning up or down or by turning off the lights when leaving a room) to protect the environment. (OECD, 2018, p. 55). Faced with this question, Charles responded:
So… my mum’s pretty strict about turning the heating down… keeping the energy use at like a certain point [because] she’s really conscious about how much money we spend on our electricity bills and our water bills. So, it’s kind of not for the purpose of protecting the environment, more from an economic… standpoint. But then I guess it’s something that’s become a force of habit. So “Yes.”
The focus of the question was to evaluate the students’ attitude towards protecting the environment. However, Charles explained that his consumption of energy is based on his mum’s push to reduce their bills. Therefore, while Charles may be recorded in PISA as “acting for collective well-being and sustainable development” in reality his attitude and actions were motivated by personal and financial concerns.
According to PISA, “global issues” are those which “affect all individuals, regardless of their nation or social group” (OECD, 2018, p.12), and so the question above assumes that all test-takers have heating or air-conditioning – or indeed electricity.For students in many of the “Global South” nations, where electricity is not guaranteed and the luxuries of heating and air conditioning are not available to many children, or for children who live in climates where these are not required, this question might not meet the criterion of relevance to everyday situations. Thus, the constructs in the assessment embed a set of values and assumptions and sets out a particular ontology of global competence which as Ledger, et al. (2019, p. 19) have pointed out “essentially describe the habitus of a global elite, making it hard to see how a child from a lower socio-economic background and/or an attendee of a poorly funded local school could possibly score well on this scale.” While PISA seeks to expand its reach into the Global South, the PISA imaginary appears to still operate within the Global North.
Choosing ethical products
Another statement to assess the students’ engagement with global issues is – I choose certain products for ethical or environmental reasons, even if they are a bit more expensive (OECD, 2018, p. 55). As with the previous example, this item could not be easily answered with just a yes/no. According to Marilyn:
Depends… I don’t… buy a lot of my own stuff so sometimes I don’t necessarily have a choice. But also… when I am buying my own stuff sometimes, I can’t really afford the… more expensive thing.
Echoing Marilyn, Marigold stated that, “I would, but I am poor.” Both Marilyn and Marigold then check “No” in response to this item. Cody also responded “No” to this statement and declared, “I’m not in charge of buying.” He explained that it was mostly his mum who was in charge of choosing things for the family.
The assumption in the statements shows that the test developers did not visualise what purchasing decisions 15-year-olds are able to make. Thus, what was intended to assess students’ engagement with global issues, resulted in an evaluation of the students’ financial situation.
What counts as a foreign language?
Another question asks students to respond to: How many foreign languages do you learn at your school this school year? On reading this question, Madeline asked, “Would this count the language I am doing out of school for VCE?” As PISA is only asking for the number of languages students are learning in school, the data will not include other avenues of language learning. For example, the survey does not take into account the complexities of foreign language courses that are sometimes taught outside of the school premises and school hours, because of the limitation in how many languages it is feasible to offer in a single school.
To the same question, Sarah responded, “Does it count, like, English?” In Australia, PISA is administered in English. Hence all references to “foreign language” in the assessment assumes languages other than English. But for Australia’s multicultural demographic, which includes newly arrived refugees, such as Sarah, and first-generation migrants in mainstream schools, English itself is a foreign language. Additionally, many other languages are spoken at home and across the student’s different environments with code-switching common in bilingual individuals (Zheng, 2009) thus making it difficult to identify what a “foreign language” is.
Trust the test?
On one hand, the assessment can be said to fulfil the psychometric requirements of validity and is therefore worthy of “trust,” but on the other hand it does not mean that we can trust the test as a representation of students’ competencies. The examples above illustrate that students’ realities are too diverse and complex to be captured by questions’ binary or reductive logic. They not only reveal how the questions can be confusing to students and how the responses might be misleading or simplistic, but also that those constructing the instruments to measure global competence appear not to have imagined the diverse nature of many parts of the world. It is also the case that in many schools, the demographic may be remarkably lacking in diversity. This is why designing a questionnaire that works across such diverse sites is so challenging – maintaining validity across multiple contexts is extremely difficult. Thus, when “real life” is reinscribed in standardised ways in these assessments, it becomes so synthesised that it becomes not-so-real after all.
PISA surveys are performative – they are highly influential and have resulted in changes to curricula, school systems, structures, and classroom and assessment practices. Given this, it is important to consider the dangers of a singular notion of the globally competent individual imagined by the small team at the OECD, becoming the model to which, all nations and all schools begin to aspire. When the diversity of contexts is sacrificed for standardised measures, there is the danger of not being able to provide locally meaningful policy recommendations. As the influence of ILSAs continues to grow, the method of survey encounters offers a counter-narrative that pleads for less emphasis on the results of ILSAs, and less faith in the “objectivity” and accuracy of such measures.
For an expert panel discussion on the PISA 2018 assessment of Global Competence, watch the webinar hosted by the Laboratory of International Assessment Studies, Global Citizenship Amid the Pandemic: Reflections on the PISA 2018 Global Competence Results. Convened by Sam Sellar, this webinar hears from Christine Sälzer, Karen Pashby and Keita Takayama. The webinar can be accessed through this link.
The Victorian Certificate of Education (VCE) is one credential that is available for students who have successfully completed Years 11 and 12 in the Australian state of Victoria.
This blog draws on two papers:
Australian Curriculum. (n.d.-b). F-10 curriculum. https://www.australiancurriculum.edu.au/f-10-curriculum/
Cardoso, M., & Steiner-Khamsi, G. (2017). The making of comparability: Education indicator research from Jullien de Paris to the 2030 sustainable development goals. Compare: A Journal of Comparative and International Education, 47(3), 388-405. https://doi.org/10.1080/03057925.2017.1302318
Deardorff, D. K. (2006). Identification and assessment of intercultural competence as a student outcome of internationalization. International Education, 10(3), 241–266. https://doi.org/10.1177/1028315306287002
Dohn, N. B. (2007). Knowledge and skills for PISA—Assessing the assessment. Journal of Philosophy of Education, 41(1), 1-16. https://doi.org/10.1111/j.1467-9752.2007.00542.x
Eklöf, H., & Hopfenbeck, T. N. (2019). Self-reported effort and motivation in the PISA test. In B. Maddox (Ed.), International Large-Scale Assessments in education: Insider research perspectives (pp. 121-136). Bloomsbury Academic.
Gorur, R. (2016a). Seeing like PISA: A cautionary tale about the performativity of international assessments. European Educational Research Journal, 15(5), 598–616. https://doi.org/10.1177/1474904116658299
Halse, C., Mansouri, F., Moss, J., Paradies, Y., O’Mara, J., Arber, R., Denson, N., Arrowsmith, C., Priest, N., Charles, C., Cloonan, A., Fox, B., Hartung, C., Mahoney, C., Ohi, S., Ovenden, G., Shaw, G., & Wright, L. (2015). Doing diversity: Intercultural understanding in primary and secondary schools. http://www.education.vic.gov.au/Documents/school/principals/management/doingdiversity.pdf
Hopfenbeck, T. N., & Maul, A. (2011). Examining evidence for the validity of PISA learning strategy scales based on student response processes. International Journal of Testing, 11(2), 95-121. https://doi.org/10.1080/15305058.2010.529977
International Baccalaureate Organization. (2017). What is an IB education? Retrieved from http:// https://www.ibo.org/globalassets/what-is-an-ib-education-2017-en.pdf
Karabenick, S. A., Woolley, M. E., Friedel, J. M., Ammon, B. V., Blazevski, J., Bonney, C. R., De Groot, C., Gilbert, M. C., Musu, L., Kelpler, T. M., & Kelly, K. L. (2007). Cognitive processing of self-report items in educational research: Do they think what we mean? Educational Psychologist, 42(3), 139-151. https://doi.org/10.1080/00461520701416231
Ledger, S., Thier, M., Bailey, L., & Pitts, C. (2019). OECD’s approach to measuring Global Competency: Powerful voices shaping education. Teacher College Record, 121(8), 1-40. https://www.tcrecord.org/Content.asp?ContentId=22705
Maddox, B. (2015). Inside the assessment machine: The life and times of a test item. In M. Hamilton, B. Maddox, & C. Addey (Eds.), Literacy as numbers–researching the politics and practices of international literacy assessment (pp. 129-146). Cambridge University Press.
Merry, S. E. (2016). Cultural dimensions of power/knowledge: The challenges of measuring violence against women. Sociologie du Travail, 58(4): 370-380. https://journals.openedition.org/sdt/915
Meyer, H. D., Zahedi, K., & Signatories (2014). Open letter to Andreas Schleicher. Policy Futures in Education, 12(7), 872-877. https://journals.sagepub.com/doi/pdf/10.2304/pfie.2014.12.7.872
Organisation for Economic Co-operation and Development. (2018). Preparing our youth for an inclusive and sustainable world: The OECD PISA global competence framework. https://www.oecd.org/education/Global-competency-for-an-inclusive-world.pdf
Oxfam. (2015). Education for global citizenship: A guide for schools. Oxfam GB. https://www.oxfam.org.uk/education/resources/education-for-global-citizenship-a-guide-for-schools
Oxley, L., & Morris, P. (2013). Global citizenship: A typology for distinguishing its multiple conceptions. British Journal of Educational Studies, 61(3), 301–325. https://doi.org/10.1080/00071005.2013.798393
Serder, M., & Jakobsson, A. (2015). “Why bother so incredibly much?”: student perspectives on PISA science assignments. Cultural Studies of Science Education, 10(3), 833-853. https://link.springer.com/content/pdf/10.1007/s11422-013-9550-3.pdf
United Nations Educational, Scientific and Cultural Organisation. (2013) Intercultural Competences: conceptual and operational framework. UNESCO. https://unesdoc.unesco.org/ark:/48223/pf0000219768
Victorian Curriculum and Assessment Authority. (n.d.). Intercultural capability. http://victoriancurriculum.vcaa.vic.edu.au/intercultural-capability/introduction/structure
Zheng, L. (2009). Living in two worlds: code-switching amongst bilingual Chinese-Australian children. Australian Review of Applied Linguistics, 32(1), 5-1.
Harsha Chandir, PhD, is a research fellow at Deakin University, Melbourne. Her research areas include international large-scale assessments, education policy and curriculum design.