Talking sense or talking power in cross-cultural assessments?

Professor Ron Hambleton of the University of Massachusetts Amherst has recently prepared a video (below) on ‘Adapting Psychological and Educational Assessments for Use in Multiple Languages and Cultures’ for the Laboratory of International Assessment Studies ESRC seminar on ‘The Challenges of Diversity’ (Lima, July 2015).  In this short blog post, Bryan Maddox writes a reply.

Ron Hambleton’s presentation describes the rise of international testing and the importance of effective test translation and adaptation practice.  He identifies ‘5 myths’ about test adaptation, describes the value of cross-cultural testing – and offers some sensible advice on what can be done to improve the quality of test adaptation.  As a world leading expert in cross-cultural testing and adaptation, Ron’s talk draws on a huge amount of experience.  We would do well, as he suggests, to study the existing literature on cross-cultural assessment such as his important contributions to the topic, and the International Testing Commission Guidelines on Translating and Adapting Tests (2010).

Ron’s talk also touches on some provocative themes and in doing so it offers to extend and enrich the debate about international, large-scale testing.  He argues that while there is a good deal known about effective test adaptation – testing programmes often lack the commitment, time and resources to implement good practice.  It seems to me that this recognition of constrained resources, time and commitment takes the debate beyond talking sense – to talking about power. In other words, it takes us beyond normative expressions of good practice to the everyday practices, interests, politics and constraints of testing programmes.  The implication of Ron’s talk and the ITC guidelines seems to be that that in order to engage effectively in test adaptation practice we need to consider more carefully how testing regimes – as political constellations of powerful actors – resolve the challenges of test adaptation and equivalence, and how they produce confidence in the validity of cross-cultural testing and comparison.

A second question provoked by Ron’s talk and the new ITC guidelines concerns the meanings of culture.  As Ron acknowledges, we should be careful in test adaptation not to assume that certain individuals (such as those who are bilingual) necessarily represent the cultural or linguistic groups involved in the test adaptation process.  Most anthropological perspectives question the idea of discrete cultures and instead note the diversity and commonality of cultural practices and values within and between social groups.  Furthermore, they take seriously the idea that power, history and flows of resources are an integral part of culture – the power to produce and sustain certain ideas, and the inequalities of power that exist between cultural groups and societies.  In that sense testing agencies are bound up in culture and cultural relations.

We might therefore ask – how do the ambitions, cultures and interests of testing regimes influence practices of test adaptation and inform validation practice? Ron Hambleton’s point is well made – a commitment to good practice in test adaptation requires us to take the time to read the ITC guidelines and to study the existing literature (which is conveniently flagged in the guidelines).  However, an additional insight provided by the growing literature on International Assessment Studies is that such a commitment to good practice also requires consideration of how power relations and interests shape adaptation practice.

Bryan Maddox, is a senior lecturer in Education and International Development at the University of East Anglia, and a director of the Laboratory of International Assessment Studies. His paper on Camels and test item fit is available here.

Photo © Néstor López. Indegenous students in Chiapas, Mexico.