Preparing for PISA in India: Skewed Priorities?

India’s first experiment with participating in the Programme for International Student Assessment (PISA) in the 2009+ cycle of testing was a disaster. The two participating Indian states, Himachal Pradesh and Tamil Nadu, finished in the bottom three places (out of 74 participating entities) for reading, mathematics and science. This was such an embarrassment for a country that was riding high on global attention due its economic success, that it quickly brushed the results under the carpet. The Government of India (GoI) refrained from participating in the subsequent rounds, claiming that Indian students performed poorly because PISA was not suited to ‘the Indian context’.

But in January 2018, the Indian government announced that it was ‘ending the boycott’ and would be participating in PISA 2021 (now to be held in 2022, due to COVID-19 disruptions to the schedule). This decision was presented as a consequence of negotiations with the OECD, with OECD agreeing to make changes to the test instrument so that ‘test takers won’t come across unfamiliar names like Bob or fruits like avocado in the test’.

Now that the test would be more closely suited to ‘the Indian context’, various officials expressed confidence that India would do very well this time around. The Chairperson of the Central Board for Secondary Education (CBSE) emphasised that India’s ‘performance in the PISA test involves the reputation and prestige of our nation’. The administration in Chandigarh, the Union Territory that will participate in PISA 2021, described the signing of the agreement between GoI and the OECD as ‘a historic day in the academic history of the country’.

Since good results in PISA are now ‘a matter of pride,’ the Indian authorities are not leaving things to chance. The government is putting in place a series of measures such as choosing the best performing territories and schools for participation (rather than taking a representative sample of the country) and engaging in intensive preparation of students and teachers. In other words, even as PISA is adapted to suit the context of India, India is adapting itself to be ‘worldclass’ and fit the image of a good performer in PISA. But this focus on good performance to bolster its reputation on the global stage could derail the ostensible objective of gaining rigorous evidence to inform policy and improve the quality of education.

Getting ready for PISA: Instant Makeover

At the international level, the OECD has also facilitated peer-to-peer learning exchanges between India and more experienced PISA countries. The Russian Education Aid Development (READ), a partnership between the World Bank and the Russian Ministry of Finance (Piattoeva & Takala, 2014), has provided Indian government officials ‘with exposure to test items, testing patterns, analysis, and reporting standards of PISA’. READ has also carried out capacity-building workshops for Indian teachers on the learning indicators and performance descriptors used for India’s National Achievement Survey (NAS) and PISA. More recently, READ has coordinated a knowledge-sharing workshop in Singapore so that specialists from India could learn about the impact that PISA policy actions and assessment systems have had on Vietnam, Singapore and Indonesia (READ, 2021). 

Sampling
For PISA 2021, of the 260 million children in 1.5 million schools in India, 5,250 students from 150 schools will be assessed on reading, mathematics and science. These children will be students of government, aided, private unaided and central government schools in the Union Territory of Chandigarh, and students enrolled in the central government school systems Kendriya Vidyalaya Sangathan (KVS) and Navodaya Vidyalaya Samiti (NVS) throughout India.

In selecting Chandigarh and KVS and NVS schools for participation in PISA, India has offered as a ‘sample’ the cream of the cohort rather than a sample that represents the nation. Chandigarh is among the highest performers in education in the nation. The KVS and NVS schools have highly selective admissions procedures and receive far greater funding and specialised institutional support compared to other government schools. Results based on testing this curated sample cannot provide information about the education system in India.

In India, an estimated 32 million 15-year-olds are no longer in school. So even if the sample of PISA test-takers were to be drawn from all school-going 15-year-olds in India, the important cohort of out-of-school children would be unrepresented in the PISA data. But India has created further exclusions which will ensure the results are not representative of even those who are in school.

Mobilising the administrative machinery

Chandigarh is a union territory and is directly administered by the central government. All KVS and NVS schools are also centrally managed and are under the control of the Ministry of Education. Taking advantage of the fact that all PISA participating entities were under the purview of the Ministry of Education, an impressive administrative machinery was mobilised to prepare students, teachers, and schools to participate in PISA.  Administrators in charge of KVS and NVS schools and schools in Chandigarh, and officials from CBSE and NCERT have all taken on key roles in preparations at the local level. These include the formation of planning committees, organising training sessions and introducing new resources. 

A monitoring cell, with teams at school, regional and national levels, has been set up to plan, execute and follow up on PISA preparations in KVS schools. In Chandigarh, a team consisting of the education secretary, the director of school education and members of subject expert committees has been set up to monitor and inspect PISA initiatives in schools. The Chandigarh administration has also constituted a further nine committees to mentor, supervise and inspect government schools as they prepare for PISA.

Doing PISA well, or doing well in PISA?

I has been argued that by preparing students for PISA 2022, a more accurate reflection of students’ learning would be obtained. But India’s preparation for PISA 2021 has, however, gone far beyond providing students with some practice to familiarise them with the format of the test. Some of the preparatory requirements listed in the KVS Teachers Handbook Preparing the Ground for PISA include the following, which arguably amount to ‘teaching to the test’:

• Students must know about PISA. Awareness in this connection must be spread among the students.
• Learning outcomes must be designed so that they cater to PISA type testingn.
• Teachers must be trained towards the PISA approachIn the timetable changes must be made to include some periods for PISA type orientation.
• Week wise and month wise worksheets can be designed to orient students towards PISA.
• Practice can be given to students on PISA based test items.Students can also be given opportunities to prepare PISA-style test items.
• Regular feedback, follow up & plan of action should be ensured at all levels and proper assessment may be attempted at regular intervals.

A frantic smorgasbord of pedagogic reforms

OECD claims that PISA is an application-based assessment with questions based on scenarios that require students to demonstrate their knowledge in novel situations. The type of learning that is claimed to prepare students for success in such assessment – described variously as inquiry-based learning, competency-based learning, 21st Century skills and so on – is frequently contrasted with rote learning, the stereotype associated with eastern school systems, including the Indian system. Participation in PISA is presented as a way for India to break free from rote learning and move towards competency-based learning and assessment.

This argument has been embraced by the government of India. Teachers are being trained in a mind-boggling alphabet soup of pedagogic approaches that broadly fall into the categories of child-centered learning and application-based learning. The KVS administration organised two-day PISA training workshops, the Chandigarh SCERT has worked with the Central Board for Secondary Education (CBSE) and international contractors to provide training for 1,500 subject teachers on the ‘experiential learning of PISA framework’, ‘pedagogy’ and the ‘formulation of PISA items in science, maths and languages’, and CBSE trained subject expert groups are training teachers in Chandigarh. Chandigarh has also introduced an app called ChalkLit to support the professional development of teachers.  Chandigarh schools have also engaged Khan Academy, an e-tutoring website, to train principals, teachers and students and In some schools one school period each week has been devoted to hands-on practice sessions  using Khan Academy materials. A list of ‘dos and don’ts’ has been given to teachers by the Education Department of Chandigarh. This list includes such items as ‘create a lively environment’ and advises teachers not to mock students who may have ‘done something silly’.

In its PISA preparation manual for teachers, the KVS administration encourages teachers to focus on the setting and attainment of learning outcomes. It is proposed that teachers introduce a number of ‘teaching styles’ including ‘spiral learning’, ‘integrated learning’, ‘experiential learning’, and ‘Preparing Students to Deal in Real Life Situation’ [sic].  The document details new assessment practices, including the integration of PISA questions into the half-yearly summative assessments and the use of regular formative assessments. 

Conclusions

Arguably, whether or not the overnight makeover of the schools participating in PISA results in better performance on PISA 2021, the attention and resources being poured into these schools will likely be beneficial to teachers and students in the immediate term. But with regard to learning from PISA, concentrated preparation may provide a skewed picture even of the school systems included in the sample, let alone the nation’s systems as a whole. The lessons from PISA may therefore not be useful – and worse, they could be misleading. However, this may not be of consequence, since neither OECD nor India are waiting for the PISA results to help determine the nature of reform required. Rather, they are proceeding full steam ahead with a range of reforms even before the test has been conducted.

If the required reforms are predetermined, it raises the question of why India should participate in PISA at all. Why not spare the anxiety, possible embarrassment, and all the expense of participation in PISA when the ‘lessons from PISA’ are already determined? Overall, India’s frantic preparations appear to be more geared towards doing well in PISA at the expense of doing PISA well to get a proper assessment of the system as it is.

It is worth noting here that India is not alone in indulging in the practice of preparing for PISA in order to get better results. Sellar et al (2019) have detailed similar exercises in Canada, Japan, Scotland and Norway, where students have been prepared and motivated in various ways to shine in PISA.

Of course, we might also ask why the OECD, which prides itself on the technical and methodological rigour of PISA, would accept India’s sampling techniques and all the preparation in the lead up to PISA that would inevitably skew results. For OECD, having India return to PISA is a major coup in its quest to expand its reach into the global south (REF). It had similarly accepted China’s initial participation with just Shanghai as part of the sample. The OECD’s motivations will form the subject of a separate blog in the coming months.

This blog is based on a case study conducted for CNESCO (Centre Nationale de Etude des systemes Scolaire).