The National Education Policy Framework launched under the government's 100-day plan calls for a number of changes to Pakistan’s educational system, such as a tech-based Smart Schools System, an Educational Volunteer Programme and an increase in the number of non-formal schools.

While the proposed reforms would be a major change to the current education landscape, they are but a new chapter in a long and active history of education reforms. If we can understand why those past programmes haven’t led to better outcomes for Pakistan’s children, we may be able to do better in the future.

Pakistan invests effort in fixing education

Over the past two decades, Pakistan has been remarkably prolific in passing education reforms. In every key area of policy, from teachers to accountability to basic inputs, the government has pushed through a major reform almost every five years.

The table below visually represents research we have done in collaboration with our colleague Jishnu Das at the World Bank on the reform experience of Punjab over a 15-year period; other provinces have shown similar levels of activity.

This pace is especially remarkable when compared to peer countries, where reforms are often subject to years of debate or blocked by powerful opposition groups like teachers’ unions or political opponents. Indeed, Pakistan has gained international attention as “perhaps the world’s largest laboratory for education reform.”

Pakistan invests money in fixing education

Vigorous activity in the policy arena has also been accompanied by a striking rise in education spending. National spending on education rose 160 per cent from 2011–2017, which fueled improvements in key educational resources like school infrastructure and teacher salaries.

Given the magnitude of changes to both policy and funding, improvements to educational outcomes seemed inevitable, but surprisingly, growth in enrollment and test scores — the two best measures of education success — has been disappointing.

There has been some improvement: today, most Pakistani children attend school at some point in their lives and more are persisting on to secondary school and university. Nonetheless, around 30pc of students do not reach sixth grade, and enrollment lags behind relatively poorer countries in Africa and Asia that have achieved near universal primary enrollment.

Learning growth is even more disheartening. Measures from the independent assessment organisation ASER show that test scores have fluctuated over the past five years and show only modest growth in the most recent round in 2018 — certainly not commensurate with the improvements in policy and influx of funding.

Why haven’t past reforms made more of a difference?

So many changes to education policy, so little to show for it: this represents a puzzle for both policymakers and academics. Analysing the effects of some of the reforms over the past two decades can reveal some of the forces that limit policy’s effectiveness.

First, unintended consequences.

Reforms can hurt just as much or more than they help. For example, the Punjab Examination Commission (PEC) exam, introduced in 2005, was intended to provide critical data on learning and equity.

However, because the exam was conducted in February, it appeared to compress the part of the school year when students are likely to learn the most: anecdotal evidence suggests that the months of concentrated learning were reduced to August through December.

Most teachers devoted January to test prep, and then had lower incentives to improve their students’ learning levels in the months following the exam, since any gains would be credited to the teacher of the next grade up.

Although we do not have the data to understand whether learning would have been greater in the absence of PEC, what is clear is that the shortening of the school year was an unintended consequence, whose ramifications are ill-understood.

In another example, several provinces have recently raised degree requirements for new teachers. While raising the bar has improved the qualifications of the average teacher, it also reduces the pool of potential teachers, which can worsen the problem of teacher shortages and high student-teacher ratios, especially in rural areas.

Opinion: Why attempts to reform Pakistani education fail

These examples are not intended to suggest that this reform should not have been implemented. On the contrary, collecting data on student learning and improving teacher quality are critical to improve learning. Rather, these examples emphasise that policymakers must carefully study the consequences of reforms and implement simultaneous policies to mitigate adverse effects.

Second, contradictory effects

When so many reforms are developed independently, they are prone to contradict each other. For example, provinces have made various efforts over the years to empower schools and parents to make decisions, such as by establishing School Councils that give parents and communities a role in deciding how some education funds should be spent.

At the same time, the government has vastly increased school monitoring and aimed to standardise school inputs, which reduces the decision-making authority of individual schools.

These efforts at decentralisation (through School Councils, school grants, contract teachers) and centralisation (through standardised curricula, monitoring, teacher hiring) stand in direct contrast and often prevent reforms on either end of the spectrum from generating a full impact.

Observing these examples, policymakers should pay greater attention to how each change is likely to interact with the broader education system. In particular, they should consider how standardisation might inhibit the ability of parents and schools to innovate and hold each other accountable, as was the intention of earlier decentralising reforms.

Piloting works

If you can imagine the unintended consequences of and contradictory interactions between policies in the profusion of reforms, then you can see why enrollment and learning gains might be slow even when certain reforms had positive effects.

The problem is, with so many hurried and overlapping changes, we don’t know which reforms have been effective and which have led to further problems. But there is a way to find out.

Before implementing reforms across the province or country, the government can try them out with a smaller group of schools. Pilot testing allows schools and teachers in the test group to provide critical feedback that can be used to improve the final policy design.

Piloting also creates the space for researchers to conduct rigorous evaluations to isolate the outcomes attributable to the specific policy in question, rather than a range of other reforms that are often introduced at the same time.

Perspective: Tackling the menace of a failed education system

Past education studies in Punjab point to the power of pilot testing. In 2004, our research group conducted an experiment to give parents and schools a “report card” showing average test scores for all public and private schools in a village.

With a careful research design, we were able to show that sharing information on learning quality led to higher test scores and enrollment and lower school fees across the communities we tested.

Drawing insights from this project, the Punjab government later decided to distribute semi-annual report cards with performance data on all public schools to help parents and School Councils hold schools more accountable for learning growth.

The current administration has a chance to take a rigorous approach to education reform, different from that taken by past governments.

After all, learning from the past is the bedrock of education.

The Analytical Angle is a monthly column where top researchers bring rigorous evidence to policy debates in Pakistan. The series is a collaboration between the Centre for Economic Research in Pakistan, Evidence for Policy Design at Harvard Kennedy School, and Dawn.com. The views expressed are the authors’ alone.