Promoting Public Knowledge of International Assessments in Education: Media and the Politics of Reception

By Mary Hamilton.

How do international assessments engage with the broader societies they are designed to serve and improve? In particular what is the discursive work done by different agencies, interest groups and the media through which the findings become part of public discourse and are translated into usable form in policy arenas?

The powerful role of international comparisons in shaping national education policies raises a number of interesting issues surrounding the role of publication, the media and more generally representation in promoting the influence of international assessments. To understand this, we need to explore how assessment findings are publicized and received. We need to find out to what extent journalists and their audiences are equipped to deal with the complex findings from international surveys and able to make informed decisions in relation to them.

We also need to consider the possibilities for exercising critical agency by intervening in these discursive processes to address the methodologies and ambitions of the surveys as well as the conflicting interests and priorities with which they have to deal.

While international comparisons in the school sector, such as the OECD’s influential Programme for International Student Assessment (PISA) are well-known, there are a range of such studies wielding their influence in all sectors of education including adult education, lifelong learning and higher education.

In our current ESRC seminar, The Politics of Reception – Media, Policy and Public Knowledge and Opinion, we examine these issues.

In his opening plenary presentation Professor Oren Pizmony-Levy will review the scope of the topic and raise key conceptual and methodological issues that will help orient ourdiscussions. Based on his own research programme he particularly distinguishes between the concepts of “public discourse” and “public opinion” and talks about empirical strategies for researching these.

To complement Professor Pizmony-Levy’s introduction I suggest we take three sets of issues as starting points for our discussions and I offer some resources to support these.

1 Mobilising individual countries to participate in international surveys

The first set of issues concern what is involved in gaining the consent and active engagement of governments and publics with the developing assessments. The international surveys coordinated by the OECD are conducted by individual countries who volunteer to take part and pay for much of the operational costs of doing so. Not every country chooses to take part in every assessment. How is access negotiated in individual countries? On what basis do governments decide whether or not to participate, weigh up the costs and benefits to them of doing so and how the assessments will fit within national contexts, both practically and in terms of the policy landscape? How is participation maintained, especially if the results do not show the country in a good light? For example, Colmant (2007) describes how France has participated selectively in international assessments of literacy and disputed the findings of the IALS in their country. Similarly the UK has not always opted in to all assessment opportunities. Addey (2013) has written about what drives lower-middle income countries to participate in international assessment programmes despite the likelihood that they will achieve low scores compared with richer countries. Using case studies of UNESCO’s LAMP programme in the Lao PDR and Mongolia, her fieldwork suggests countries join these programmes as part of what she calles “a global ritual of belonging”. Other countries, in contrast may value the “glorification” (Steiner-Khamsi 2003) of their education system offered by high positions in the international league tables.

In the seminar our discussion of these issues will be informed by a presentation from Dr. Sarah Richardson, now Australian Council for Educational Research in India who will be talking about her experience of directing the feasibility studies for the AHELO programme in a range of different countries, especially issues around motivation and the varying responses of those countries to the idea of assessing HE student learning outcomes. Catherine O’Connell provides a background briefing paper discussing the success of the university rankings and comparing these with the experience of the AHELO initiative. AHELO is interesting for the purposes of the seminar because it is an initiative that encountered considerable institutional resistance unlike the popular global university rankings. A similar contrast can be seen between PISA which seems to go from strength to strength despite critiques, and PIAAC (the assessment of adult skills) which seems to have attracted little attention so far.

2 Managing the Public Release of findings from international assessments

Once the survey data has been collected and analysed, the release of the findings has to be managed, both internationally and within individual countries. The issues involved in this process concern the communication strategies of international agencies such as the OECD, UNESCO, or the IEA (the International Association for the Evaluation of Educational Achievement). These agenciespresent a range of carefully crafted artefacts to the media and to national stakeholders. Stakeholders include advocacy groups, policy makers and their research bodies, commercial organizations, academics, and ultimately parents, students and the general public.

At the Lancaster seminar, these issues and more will be addressed by a panel of contributors. A speaker from the UNESCO Institute for Statistics in Montreal will talk about the strategic approach of an international agency. Dr Megan Knight, University of Hertfordshire, will talk about the changing professionalism of media workers particularly the growing specialism of data journalism which responds to the complexity of the international assessments (see also Rogers, 2016 who created the UK National Guardian datablog). Dr Petra Javrh, Slovenian Institute for Adult Education will describe the approach of a government advocacy agency working with journalists to prepare for the release of the PIAAC findings. Dr Cormac O’Keeffe from YES N’YOU, a language teaching school in Paris, will talk about the statistical workshops offered by the OECD to train specialists in participating countries to to carry out further analysis of the PIAAC findings. Professor Heinz-Dieter Meyer reflects on the value and effectiveness of academic activism from his experience of intervening in the public discourse of PISA through an open letter to Andreas SchleicherHis letter, signed by a large group of academics and education professionals, was widely reported in the international media and provoked a response from the OECD. We will discuss what changes resulted from this and what we can expect from such interventions.

3 Researching and intervening in how the findings are reported and interpreted in the media

Dr Javrh’s work leads into our third issue: how the findings are actually reported by the media and the various interpretations attached to them within public and policy discourse. What presentational formats have been developed to communicate the findings, including both narrative and a range of visualizations of the numerical findings through images, tables and diagrams? (See Hamilton 2012; Williamson, 2015). How are the findings publicized across the variety of interlinked media platforms now available – newspapers, broadcast media, online news sites and social media, especially Twitter. Dr Aspa Baroutsisand Professor Bob Lingard will address this aspect of the politics of reception, reporting on their recent study of the Australian press and PIAAC. Other recent studies can inform and extend this discussion including Cort and Larson, 2015; Mons and Pons, 2009; Rubenson and Walker, 2014; Waldow et al, 2014; Yasokawa et al, 2016.

Taken together, these issues offer a rich seam for discussing and theorizing the crucial topic of how survey findings are translated into policy and practice.

 

Annotated Bibliography

The references listed below are intended as a resource for participants and others to add to and to extend the discussions in the seminar itself. Some of the authors will also be present at the Lancaster seminar.

Mobilising individual countries to participate in international surveys

Addey, C. (2014). Why do countries join international literacy assessments? An actor-network theory analysis with case studies from Lao PDR and Mongolia (Doctoral dissertation, University of East Anglia). https://ueaeprints.uea.ac.uk/52185/1/2014AddeyCCLPhD.pdf

Coates, H., & Richardson, S. (2012). An international assessment of bachelor degree graduates’ learning outcomes. Higher Education Management and Policy, 23(3), 1-19.

Colmant, M. (2007) The Impact of PIRLS in France. In Schwippert, K. (ed) Progress in Reading Literacy, Verlag. p 75

O’Connell, C (2015a) Close up examination of discourses associated with Global University Rankings: counter-narratives in the UK context. Higher Education Quarterly 69 (3) 279-294.

Richardson, S. (2015). Cosmopolitan Learning for a Global Era: Higher Education in an Interconnected World. Routledge.

 

Managing the Public Release of findings from international assessment

Grek, S. 2010. “International Organisations and the Shared Construction of Policy ‘Problems’: Problematisation and Change in Education Governance in Europe.” European Educational Research Journal 9 (3): 396-406.

Meyer, H. D., & Benavot, A. (Eds.). (2013). PISA, power, and policy: The emergence of global educational governance. Symposium Books Ltd.

Pizmony-Levy, O. (2013)Testing for all: The emergence and development of international assessment of student achievement, 1958-2012 ISBN: 9781303482786 http://gradworks.umi.com/35/99/3599130.html

Rogers, S. (2016) Data journalism matters more now than ever before http://simonrogers.net/2016/03/07/data-journalism-matters-more-now-than-ever-before/

Steiner-Khamsi, G. 2003. “The Politics of League Tables.” JSSE-Journal of Social Science Education 2 (1). DOI http://www.jsse.org/jsse/index.php/jsse/article/download/470/386

 

Researching and intervening in how the findings are reported and interpreted in the media

Cort, P., & Larson, A. (2015). The non-shock of PIAAC–Tracing the discursive effects of PIAAC in Denmark. European Educational Research Journal, 14(6), 531-548.

Hamilton, M. (2012) Literacy and the Politics of Representation. London & New York: Routledge.

Javrh, P. Manninen, J., Sgier, I., Fleige, M., Thöne-Geyer, B., Kil, M., Možina, E., … & Diez, J. (2014). Benefits of Lifelong Learning in Europe: Main Results of the BeLL-Project. Bonn: DIE.

Knight, M. 2015. Data journalism in the UK: a preliminary analysis of form and content. Journal of Media Practice 16 (1): 55-72.

Mons, N., X. Pons, A. van Zanten, and J. Pouille. 2009. The Reception of PISA in France. Connaissance et régulation du système éducatif. Paris: OSC.

Rubenson, K., and J. Walker. 2014. “The Media Construction of an Adult Literacy Agenda in Canada.” Globalisation, Societies and Education 12 (1): 141-163.

Takayama, K., Waldow, F., & Sung, Y.-K. (2013). Finland has it all? Examining the media accentuation of “Finnish education” in Australia, Germany, and South Korea. Research in Comparative and International Education, 8(3), 307-325.

Waldow, F., Takayama, K., & Sung, Y.-K. (2014). Rethinking the pattern of external policy referencing: Media discourses over the “Asian Tigers’” PISA success in Australia, Germany, and South Korea. Comparative Education, 50(3), 302-321.

Waldow, F. (2016). Das Ausland als Gegenargument: Fünf Thesen zur Bedeutung nationaler Stereotype und negativer Referenzgesellschaften. Zeitschrift für Pädagogik, 62(3).

Williamson, B. (2015). Digital education governance: data visualization, predictive analytics, and ‘real-time’policy instruments. Journal of Education Policy, 1-19.

Yasukawa , K. Hamilton, M. and Evans, J. (2016) A Comparative Analysis of National Media Responses to the OECD Survey of Adult Skills: Policy Making from the Global to the Local? Compare http://dx.doi.org/10.1080/03057925.2016.1158644

 

Blog photo copyright: Stougard_2008