As we struggle with Covid-19, when it comes to schooling, we are all asking the same questions — will we finally get back to normal? And is normal where we want to be, or is it time to change everything, starting from the ground-up?

But even as we ask these questions, here is a little secret: Despite the attention that schooling and learning have received in Pakistan, and internationally, we really don’t know what ‘normal’ is.

We know some things — for instance, children in many countries cannot read a full sentence or add two-digit numbers after spending 5 years in school. But for anything harder, we are still in the dark. We still don’t know how much children learn during primary school. Neither do we know whether children who are initially behind fall even further behind as they progress through school.

To address these problems, Jishnu Das, Tahir Andrabi and Asim Ijaz Khwaja started the Learning and Educational Achievement in Pakistani Schools (LEAPS) project in rural Punjab in 2003.

The LEAPS data follow more than 12,000 children as they progress from 3rd to 6th grade and are among the first data from a low-income setting to allow us to examine children’s test scores in a consistent manner across four years.

Our paper , jointly written with Andres Yi Chang, uses the LEAPS data to finally benchmark what normal means for a country like Pakistan.

Here is what we learned.

First, not surprisingly, children do learn in school. For instance, 58 per cent of children could correctly multiply “4 x 5” in grade 3, and this fraction increases to 60pc after a year, 73pc after two years, and 79pc after three years.

We see similar patterns across every question and subject and, on average, a child in grade 6 knows more than 77pc of children tested in grade 3. This rate of learning is similar to what we find in Vietnam, Peru, India and Ethiopia and also to the US state of Florida.

In all these school systems, the top 30pc of children in grade 3 score (roughly) the same as the bottom 30pc of children in grade 6. This, however, does not imply that they learn the same amount since the tests and initial learning levels are different across countries; the data to answer that question simply do not exist.

Second, policymakers in Pakistan have been deeply concerned about out-of-school children. To understand the link between learning and dropping out of school, the LEAPS data tracked and tested children who dropped out between grades 5 and 6.

Surprisingly, we found that children who eventually dropped out in the transition to middle school were learning just as much as those who had continued (even though in every year, their test scores were slightly lower).

Further, once children dropped out, their learning stalled, while for those who remained in school, it continued along the same trend (Figure 1). So, a narrative that suggests that children drop out because they are not learning is not supported by the data.

Third, we examined whether children who were performing worse in grade 3 fall farther behind. Figure 2 shows that this is not the case by grouping children by how much they learnt between grades 3 and 6 (from low to high) and showing their average test score in grade 3.

In fact, children whose test scores were in the bottom 10pc in grade 3 learned significantly more by grade 6 than children ranked in the top 10pc learners. The same happens across the other groups which suggests that schooling reduces inequality in learning.

To understand how these facts can still be consistent with low test scores, we then measure the patterns of gains and losses on a question-by-question basis.

We find that test scores do not increase every year. In fact, 20pc of children see declines in test scores on a yearly basis, and 10pc reported lower test scores in grade 5 compared to grade 3.

We propose a new term for these patterns in the data: ‘fragile learning’.

Children learn in one year but are about as likely to forget as to consolidate their learning. In fact, the proportion of ‘fragile learners’, or those who learn and then forget is worryingly high. The key message is that performance in school has as much to do with forgetting as it does with learning.

Read: How a multilingual education model can help Pakistan increase literacy

One of us grew up in a very similar schooling system where the emphasis was always on rote learning. We always knew that rote systems did not address conceptual weaknesses, but if there was one thing that such a system is meant to do, it was to ensure that children never forgot a question they could answer correctly once.

Perhaps weak conceptual understanding drives this phenomenon of ‘fragile learning’, but the fact of the matter is that despite hours spent in rote learning, children still can’t remember what they learnt in the previous year.

We now know what ’normal’ means. It means that children who go to school learn. It means that children who are dropping out leave school for a host of reasons, but not just because they were learning less. It means that our schools are an equalising force in these children’s lives — those who start off knowing less end up learning more as they progress through school, though they may not fully catch up.

But it also means that our current systems seem to be encouraging a ‘learn and forget’ rather than a 'learn and consolidate' approach, at least for a sizeable minority.

Our plea therefore is to not label all schooling as poor or useless. The fact of the matter is that a very important part of the schooling system — the pedagogical approach that leads to fragile learning — needs reform, now more than ever, as children return after a long absence.

But there are things that are working and we should be careful not to throw the good out with the bad as we start the journey to a system where every child can truly learn.

The Analytical Angle is a monthly column where top researchers bring rigorous evidence to policy debates in Pakistan. The series is a collaboration between the Centre for Economic Research in Pakistan and Dawn.com. The views expressed are the authors’ alone.

Natalie Bau is an Assistant Professor at University of California, Los Angeles and Jishnu Das is a Professor at Georgetown University, Washington DC.

Header photo: Children wearing facemasks attend a class at a school in Lahore. — AFP/File