Evaluation in an interconnected world
Systematic evaluation can help us identify workable strategies to address the pressing problems the world is facing, but it has to raise its sights higher
COVID-19, the novel coronavirus pandemic that has been estimated to have infected some 700 million people and killed more than 7 million around the world, should have been a wake-up call. The economic and social consequences were severe, as countries scrambled to contain the spread of the virus. The virus SARS-CoV-2 that caused COVID-19 crossed over from its non-human host to a human in or around the city of Wuhan in China in late 2019. The exact transmission mechanism is still not known but the root causes are clear. The spill-over of zoonotic viruses like SARS-CoV-2 is becoming more common as we come into ever closer contacts with animals, both domesticated and wild. As human activities extend deeper into undisturbed ecosystems, undiscovered pathogens are released. The destruction is driven by the expansion of agriculture and cattle ranching, logging and deforestation, road construction, mining, new settlements and urban sprawl, making space for the growing human population and its ever-increasing demands for raw materials, food stuffs and consumer goods.
These are the very same factors that drive other complex global problems, including climate change, biodiversity loss, and pollution that continues to kill millions of people every year. It also shows how interconnected today’s world is; how economic development and environmental degradation are intimately intertwined. As we cut down trees, not only do we come into contact with lethal pathogens, but we also undermine the forest’s ability to sequester carbon thereby speeding up global warming. As people get richer, their diets tend to become more meat-based. There are now half a billion cows and 23 billion chicken on the planet. It is said there is a patch the size of Denmark in the Amazon that has been cleared to grow soy beans to feed pigs in Denmark. At the individual level, a consequence of the increased meat consumption is higher rates of obesity, diabetes and cardiovascular diseases even in countries that previously didn’t experience them. Human health and ecosystem health are inseparable.
Furthermore, these global problems impact different groups and communities differently. Climate change affects the poor and vulnerable communities hardest, whether it is those living on the low-lying coast of Bangladesh pummelled by more frequent cyclones and sea-level rise or small farmers in African drylands suffering during prolonged droughts. Similarly, the pandemic’s impacts were felt most acutely by the poorer segments of society and minorities who could not afford to shelter in place.
What does this all have to do with evaluation? Everything, in my opinion, because evaluation must concern itself with the real problems that affect people’s lives and livelihoods.
As humans, we evaluate constantly and intuitively when we make choices in our daily lives, whether it pertains to small matters (what to have for dinner) or big things (where to go to school). However, evaluation is also a profession and a field of applied research with its own traditions, rules and practices. In a widely accepted formulation, OECD defines evaluation as: “The systematic and objective assessment of a planned, ongoing or completed intervention, its design, implementation and results. The aim is to determine relevance, coherence, effectiveness, efficiency, impact and sustainability. Evaluation also refers to the process of determining the worth or significance of an intervention. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into decision-making processes.”
I would argue that evaluation practice is too narrowly focused on specific interventions to effectively respond to these global challenges. Today, many evaluators write about these issues using terms like ‘complex’ and ‘wicked,’ but I am not sure that the practice of evaluation has kept up with the theory. Evaluation as a profession has its roots in social inquiry, where we test the effectiveness of interventions on a well-defined treatment population against a control group. We may use experimental or quasi-experimental tools, or we may lean towards more participatory and qualitative approaches, but either way the focus is mostly on a single intervention and its effects. Evaluations test the effectiveness of the intervention in terms of its pre-determined objectives. The desire is to be able to attribute any changes in the outcome to the intervention—or, recognizing the complexity and presence of multiple actors, at least the specific contributions of the intervention.
Apart from being narrowly project-focused, evaluations are driven by donor concerns for accountability and ‘value for money.’ This treats the central question as a matter of simple accounting instead of a choice between strategies that would address the root causes of the problems we face. The accounting is more often than not for the purposes of the funders and their priorities, not for the benefit of the claimholders that the intervention is intended to benefit.
Seldom do evaluations look at the big picture: Are we actually doing the right thing in the broader system landscape? Is the chosen strategy effective? Is the intervention that we are promoting meaningful in the larger whole? Is it something that the intended beneficiaries want and need? Is it fixing one part of the problem but creating others elsewhere? Is it having unintended consequences for the environment, for women, for disadvantaged groups, for Indigenous peoples, for power relations, etc.?
We must incorporate the environment into our evaluations. Sustainable development lies on social, economic and environmental foundations, yet evaluation—like national accounting—is almost exclusively concerned with the economic and, to a lesser degree, social capital, while natural capital and its depreciation are considered external to the system. According to the World Bank, low-income countries get 47% of their wealth from natural capital. This figure certainly underestimates the value of ecosystem services, in terms of clean water and air, health benefits, protection against natural hazards, recreation etc. Evaluators must learn how to operate at the nexus of environment and development, which means understanding the interplay between human and natural systems.
For evaluation to remain relevant, it must rise above its project mentality and start looking beyond the internal logic of the interventions that are evaluated. It must systematically search for unintended consequences that may lie outside of the immediate scope of the evaluation. It must expand its vision to encompass the coupled human and natural systems and how they interact. And it must resist focusing on accountability for funders and instead make sure that it contributes to learning, for the wellbeing of people and nature in an equitable manner. If we achieve this, evaluation will be better positioned to contribute to more sustainable and just development in an interconnected world.