measurement

The best way to measure a household’s resilience? Ask those who live there

Article published origionally at the Guardian

7887050306_7552b01753_z.jpg
Image: Internews Europe

A new study challenges the traditional assumption that experts are best placed to evaluate how well people could cope with and adapt to an emergency

The concept of “resilience” is taking development and humanitarian sectors by storm. Huge amounts of finance are being channelled into “resilience-building” activities, aimed at supporting people and communities to deal more effectively with climate extremes, financial shocks and the many other risks that threaten lives and livelihoods.

Given the pressure on NGOs, governments and donors to demonstrate value for money and support the right people and activities, the race is on to find the best ways of measuring resilience.

Normally, the first step in designing a method for resilience measurement is to consult a group of experts, who consider the assets and capacities that make a household robust. They typically come up with a number of indicators, which can consist of anything from household income and child nutrition to social networks and access to financial capital. Each indicator and characteristic is then mashed together and weighted, often resulting in a single overall score.

While useful, these approaches make a critical assumption: that experts are best placed to evaluate someone else’s resilience. In a new paper for the Overseas Development Institute, we challenge that assumption and propose an alternative approach that has been largely overlooked, but may help to address many of the challenges associated with traditional ways of measuring resilience.

The approach is called subjective resilience, and it starts from the premise that most people have a good understanding of the factors that contribute to their own ability to cope with and adapt to emergencies.

People are asked to consider the factors contributing to their livelihoods and judge how resilient they consider their household to be to given threats. They are also asked to suggest ways to enhance their resilience.

To test the approach, a consortium of international NGOs and research organisations under the Global Resilience Partnership teamed up with Twaweza, an east African movement working to give citizens more power, to speak to 1,300 households across Tanzania, asking people to evaluate their ability to cope with and adapt to the risk of future flooding.

The method used to collect data was also quite unique. Face-to-face interviews were done with people in households from the shores of the Indian Ocean to the Serengeti. Each person was then given a mobile phone and a solar charger, and every few months a call centre rang the same people to ask a series of questions. The answers were then analysed to give accurate and nationally representative information and the whole process was cheaper and easier than traditional surveys.

Given that surveys relating to subjective resilience tend to be shorter than objective ways of measuring resilience (which can often be more than 100 questions long), the approach lends itself to data collection of all kinds across Africa.

Initial results suggest that many of the factors that experts associate with a person’s resilience may not be as important as they had assumed. Factors such as level of education, livelihood, or age have some effect on people’s perceived resilience, but relationships are not as strong as one might traditionally expect. Other factors, like gender, have no apparent impact on evaluations of household resilience, even among female-headed households (typically considered to be one of the most vulnerable groups). Unsurprisingly, the strongest predictor of resilience is wealth and level of income.

While the research is in its early stages, and more work needs to be done to test and develop subjective methods, what we have seen so far raises questions as to whether the indicators we equate with resilience are the right ones.

Above all, subjective assessment has the potential to radically change the way that we track resilience and hold governments and civil society to account.

It allows for a bottom-up way of judging the effectiveness of resilience-building initiatives based on the perspectives of the people that matter most: those who are vulnerable and receiving support. One interesting finding from the Tanzania survey was that those who received early warning information about floods rated themselves as far more resilient than those who hadn’t. This suggests that the considerable efforts to support dissemination of weather forecasts and alerts in east Africa may be paying off.

This approach could be used to assess the effectiveness of investments and projects from the perspective of the people benefiting from them. If we listen to their collective voices, we might be better able to hold NGOs, businesses and governments to account.

The same tools could also be applied to evaluate national or international resilience-building initiatives: a difficult feat so far. If international commitments, such as the sustainable development goals, are taken seriously and policies implemented to achieve them, then it is only reasonable to expect a marked difference in how resilient local people perceive themselves to be.

It is clear that subjective assessments are not without weaknesses. People aren’t always aware of all the factors that make them resilient, and some may choose to answer questions inaccurately in order to gain from the outcome of the survey, particularly if there is an assumption that the survey is linked to NGO or government assistance. However, many of the biases can be accounted for by thorough research design and by providing clear information on how the data will be used. Confidence can be taken from the success of similar approaches in other fields, such as subjective wellbeing.

Ultimately, collecting information on subjective resilience isn’t meant to replace traditional measurement entirely. Rather, bottom-up subjective methods should be used alongside objective methods, helping to capture the components of resilience that are difficult to observe and allow people’s perspectives to be heard.

If we can get the measurement process right, this will be an important step forward in gaining a more holistic understanding of what it takes for a household to be resilient to the many risks and threats it faces.

Are we jumping the gun in trying to measure adaptation?

Article originally published at ODI Blogs

14999534034_ba01564b36_z.jpg
Image: ▓▒░ TORLEY ░▒▓

A clear and concise methodology for measuring adaptation to climate change has yet to emerge. But the need for it is clear.

With the impacts of climate change threatening to undermine development objectives and substantial pots of money being committed to support adaptation, evaluating the impact and effectiveness of adaptation interventions is paramount. The UK government for one has committed £2.9 billion for 2011-2015, a balanced proportion of which is thought to be earmarked for adaptation, and many other countries have committed through so-called ‘fast-start climate finance’.

Why is there no agreement?

To start with, we have to realise that assessing and understanding adaptation, and the processes that shape it, is incredibly complex.

Ahead of the fifth Community Based Adaptation (CBA5) conference in Dhaka, Bangladesh on 28 March, I focus here on measuring the adaptive capacity at the local level. A number of useful early attempts have been made– most maintaining strong links with the sustainable livelihoods framework (SL) and its five capitals (human, financial, social, natural and physical), with many simply measuring the abundance of each capital as direct indicators for adaptive capacity. These make the assumption that the principles of sustainable livelihoods are identical to those needed to adapt to a changing climate.

However, it’s important to realise that there is no agreement about the characteristics and indicators that support adaptive capacity. While true that the SL’s five capitals are, to a large extent needed to facilitate adaptation, this does not give a complete picture of the complex processes that determine adaptive capacity and support adaptation actions.

So have we been too keen in trying to measure something that we don’t yet fully understand?

In some ways the answer is yes, as little empirical research has actually been done to characterise adaptive capacity. This is where the Africa Climate Change Resilience Alliance (ACCRA) is trying to add clarity to the debate. The project looks to use evidence-based research to explore the characteristics of adaptive capacity at the community level, using Ethiopia, Mozambique and Uganda as case studies. As part of this, we at ODI have reviewed existing methods and built on pilot consultations within the three countries as part of the ACCRA consortium to present the Local Adaptive Capacity framework (LAC).

Preliminary findings from the project point to five common characteristics of adaptive capacity, which make up the core of the LAC framework. These are: a diverse asset base; appropriate and fair institutions and entitlements; access to relevant knowledge and information; an enabling environment that fosters innovation; and flexible forward-looking governance and decision-making processes. These characteristics are thought to be conducive to supporting local communities in enhancing their adaptive capacities. The conceptual basis behind these five characteristics and their applications are explored in further detail in a recent ODI report.

The first thing that’s striking about the five characteristics is that many of them are actually processes, and not simply assets. This contrasts with traditional methods of measuring adaptive capacity and suggests that solely focusing on the SL’s five assets as indicators is unlikely to provide a holistic view of adaptation. It is therefore important to realise that adaptive capacity is as much about what a community does that enables it to adapt, than what a community has that allows it to adapt. The reason that this distinction is important is because how we measure adaptation will ultimately guide how we design interventions aimed at supporting it, and may skew the focus of project interventions.

What can we learn from ACCRA’s research?

Most importantly, perhaps, is that before we dive head-first into trying to monitor and evaluate adaptation at the community level, we must ensure that we base our M&E on a firmer understanding of the characteristics and indicators of adaptive capacity. This would also enable us to see how certain wider development interventions – many of which may not have been designed with climate change in mind, such as social protection or livelihoods programmes – are contributing to characteristics of adaptive capacity. Fortunately, a number of other research institutes are complementing ODI’s work on ACCRA, and attempting to step back and explore these issues in more detail. Through efforts like these, we are gathering a stronger evidence base for measuring adaptive capacity at the community level, and understanding the importance of processes in supporting adaptation. Reassuringly, this knowledge and information not only has uses in refining M&E indictors, but it can also pave the way for better operationalisation of programming for NGO, civil society and policy-makers.

These actions will prove a useful step forwards in improving our understanding of complex process of adaptation, and ultimately aid the delivery of assistance to those most in need and most vulnerable to the impacts of a changing climate.

How can we measure resilience? Mobile phones – and the right questions – can help

Article published origionally at Thompson Reuter Alert Net

2736558918_875f87479a_z.jpg
Image: Erik (HASH) Hersman

Phone surveys and efforts to ask people what they think makes them resilient may change some old ideas

People and communities around the world are struggling to deal with the impacts of climate extremes and disasters. At the same time, international finance for supporting people’s resilience to shocks and stresses is limited.

That means understanding how to effectively build resilience is crucial – but to do that we first need to be able to track and measure resilience – something that is often fiendishly difficult.

For example, we might consider a resilient household to be one that can take precautions after receiving early warning of an imminent flood; bounce back quickly from a recent drought; or adapt to increasingly frequent heatwaves. But deciding what factors contribute and are most important to a household’s resilience is a matter fierce debate. Dozens (if not hundreds) of different resilience frameworks exist, each with a unique mix of indicators and ideas.

To make matters worse, collecting information on resilience is hard work. Face-to-face household surveys are expensive, time consuming to run and can take months to set up. This is where the Building Resilience and Adaptation to Climate Extremes and Disasters (BRACED)’s Rapid Response Research (RRR) is making a difference.

RRR is a survey effort that collects information on resilience and post-disaster recovery, currently focusing on the east of Myanmar (in the township of Hpa-An). The initiative is trying two new methods that have the potential to drastically change the way that we collect resilience information.

The first is the use of mobile phones to gather information from households affected by disasters.

With a rise in mobile phone usage across the developing world, contacting people and collecting data has never been quicker, cheaper or more secure. As part of the RRR effort, 1300 mobile phones and solar chargers were given to households across eight villages in Hpa-An. A call centre based in Yangon then administers short surveys by phone once a month, with households receiving a small financial reward – in the form of airtime credit – for every survey they complete. If households are busy, they’re simply asked for a preferred time to be called back.

This means that not only can we can collect data at roughly a third of the cost of traditional surveys, but it can also be collected when people are on the move. This is necessary in a place like Myanmar where people are increasingly mobile – often seeking temporary work in cities and abroad. Crucially, it means that we have an easier (and less intrusive) way of contacting people after disasters, when gaining access to communities can be slow and high risk. This could be especially useful in instances where people have relocated after a disaster, which would not be possible through normal survey methods.

So far, these methods have allowed the RRR survey to retain 96 percent of the original survey respondents after four separate rounds of surveying. That’s a number that has far exceeded expectations!

The second innovation trials new ways of judging subjective measures of resilience. Resilience has traditionally been measured via objective means – where resilience ‘experts’ come together and decide on a list of indicators that they think make people resilient. This typically includes things that we can see and observe such as household income, education, access to social safety nets, etc.

While methods like these are no doubt useful, they struggle to capture many of the intangible aspects of resilience, such as social networks. Subjective tools, like the ones the RRR effort is trialling, take a very different approach. They start from the position that people have valuable knowledge about what they think makes them resilient.

What have we found so far? Subjective views of resilience are strongly associated with education, poverty, number of household occupants and so on. While traditional assessments reflect many of these, a number of interesting differences exist with objective assessments of resilience.

For example, female-headed households in Hpa-An think of themselves as better able to deal with disasters compared to households headed by men. This flies in the face of many objective surveys that tend to find male-headed households more resilient.

Could it be that female-headed households are able to leverage better social support networks, or tend to have more diverse sources of livelihood pursuits? Could it instead be that there is a psychological difference in how women and men rate themselves? These are questions that the RRR will delve into in the months ahead.

The RRR effort continues to collect large swathes of data. To make this information accessible to all we’ve launched the Resilience Dashboard. This site allows anyone to look in real time at the relationships and trends for themselves.

We hope to learn from those making use of the site to see what potential this new technology and method has, as well as what new ideas it can spark. Above all, we want RRR to generate enthusiasm about innovating and experimenting with different ways of collecting resilience information to help further our understanding of the drivers of resilience.

Only then will we be able to answer the important question: How do we best prioritise limited resources for supporting resilience?