EMPIRICAL evidence is a cornerstone of public policy because it enables informed decision-making. Embedding the world of research within the world of public policy can reap significant benefits by arming policymakers with important information to tackle development challenges. At the same time, the worlds of research and policy are different which means that embedding both also leads to important concerns that need to be resolved.

Let’s explore the first type of concern with an illustrative example.

Assume that you are a policymaker who wants to introduce cash-transfer programmes in the country with the objective of reducing economic poverty. Rather than introducing the programme across the country, you decide to pilot it in a small town and partner with researchers to run an impact evaluation. The researchers suggest running a randomised controlled trial — which is a fancy way of saying that they would compare some people who would get the cash transfers with others who would not (there is some randomisation involved as well).

The policymaker has, however, a tight timeline and wants to decide whether this programme can be scaled up in a few months. Theme researchers on the other hand caution that for them to confirm the true impact of the policy (also known as internal validity), they might need a few years. Ideally, they would want to go through the peer review process to ascertain whether or not the policy is effective.

Interaction between policymaking and research throws up concerns.

These differing incentive structures of policymaking and research create concerns that need to be resolved. The world of policy often requires quick decision-making while academic empirical research puts a higher premium on getting the right answer regardless of the timeline.

The second concern relates to the wider applicability or generalisability of the evid­ence (also known as external validity). Assu­­­me now that the researchers have ascertained that the cash transfer programme is indeed effective in reducing economic poverty.

One important question that remains to be answered is whether the impact would be similar if the programme was scaled up nationally. Some important questions to answer could be: would the implementation of the cash-transfer programme become more or less difficult if it is scaled up for the whole country? Does state capacity to implement projects differ across regions in the country? Does the differing local context across regions make implementation easier or harder?

As a policymaker, how do you approach resolving these two concerns of differing timelines and generalisability? Fortunately, there are various approaches to address the problem.

Starting with the first concern about timelines, policymakers have to strike the right balance between internal validity and timeliness. While it might make sense to run a long-term study on the impact of a policy, researchers can also rely on other less time-intensive (though less internally valid) methods that can give policymakers important information about the likelihood of effectiveness.

Researchers can also provide preliminary findings of their evaluations which might not be peer-reviewed but would still give decision makers an informed view. At the same time, it is important to still have flexibility within the policymaking framework to incorporate findings as and when they come out. Say, for instance, that all preliminary findings reveal that the programme is likely to be effective, but longer-term impact evaluations reveal that there may be some unintended consequences of the policy. In this case, it would be important for the policymaker to be flexible in changing the programme based on these findings.

The second concern of generalisability can be trickier to handle. Here, policymakers can use a combination of theory and evidence to judge whether the policy is likely to work in a similar manner if scaled up. For instance, if the capacity of the civil service remains high in certain geographic areas of the country, uniform implementation of the policy in these regions might be less of a concern. It would also be important for policymakers to think about how the underlying mechanism/ theory of change would interact with the local context. All of this can of course not be estimated ex-ante, which again means that it is important to have a continuous feedback loop on implementation so that policymakers can adjust the programme quickly.

Embedding the world of research within that of policymaking is crucial for improving development outcomes. At the same time, merging these two worlds also means that policymakers and researchers need to address the two concerns of differing timelines and generalisability. The wider society stands to benefit immensely if these tensions are addressed in an effective manner.

The writer has a doctorate from the University of Oxford and is graduate of the Harvard Kennedy School of Government

Twitter: @KhudadadChattha

Published in Dawn, March 2nd, 2022