Data dive: Measuring poverty through real-time data

Crossposted from the Voices from Eurasia Blog by UNDP

Looking at the many dimensions of poverty, one can realise what a highly complex issue it is. The monetary perspective is just a start-off, there are also many other relevant areas such as health, acccess, location, environment, mobility, etc. Amartya Sen has done a groundbreaking work and much progress in recent decades to measure poverty. But one big problem persists: Poverty analysis is backwardly focused. To put it in blunt words, we can measure how it happened, but it is very difficult to measure it as it happens. Statistics are often months or even years old. When people apply for social welfare the personal crisis has already occurred. But how can the symptoms be measured before they even happen?

Such a new approach is to use real-time data (or big data) to measure poverty as it is happening. As we enter an era of massive data collection, thanks to the Internet and mobile devices, how could such data be also used to understand many dimensions of poverty? One word of caution : Better analysis can lead to better action to fight poverty, but it does not take people out of poverty by itself.

Data dive event Vienna

This new promise is a starting point for a series of data dive events organized by UNDP and the World Bank in the next months. One event already took place on February 22nd organized together with the Open Knowledge Foundation Austria, where I had the pleasure to facilitate the event. We were happy to have 26 participants from various backgrounds such as data analysis, spatial analysis, visualization, crowdsourcing, business intelligence, data protection and programming. For the event various data sources were provided thanks to the generous contribution of TextToChange and the Qatar Foundation.

The overall good turn-out of participants, for such a niche topic, shows the high interest in big data analysis and proves the potential to reach out to external resource persons. It is promising to see data specialists investing their free time to participate on such events. Since big data analysis is in its infancy within development organizations, this is a chance to combine inside expertise with outside data specialists. Personally, I think it is particularly helpful to invest in a community for that topic, instead of relying on business intelligence solutions.

My first observation during the discussion was a bit of a clash of mindset between classical data collection for statistics and new opportunities with big data and data mining approaches. How can these two approaches can be used together? How can one support the other for better poverty analysis? But that did not stop participants from brainstorming interesting ideas to measure vulnerabilities.

Various ideas

One idea was to look at data from retail chains to track consumption patterns to analyze the risk of poverty on a regular basis. For example, having exact figures from recent weeks how certain commodities have been purchased. How staple food is maybe replaced by more luxury and vice versa. Bonus-points-collecting services have fine granular data for millions of consumers, where larger and smaller shifts in consumption patterns can be analyzed and localized in real-time. Maybe this data could be used once for a good cause, instead of tweaking the last bit from it to identify consumer interests for marketing purposes. But to what extent people affected by poverty, particularly in developing countries, are tracked with their consumption behaviors?

Another approach is data from insurance companies, which can be used as an early indicator for vulnerabilities. Particularly insurance companies have long tradition for risk assessment, and through micro-insurances their data reaches far into rural areas also in developing countries. A participant proposed measuring the distance for daily commuting and how this might change due to economical constraints. Or energy providers could provide data for payments as an early indicator when electricity or heating bills cannot be paid. If one gets access to companies' data, the flow of remittances could be analyzed in real-time to see larger and smaller trends of money transfer and what it might indicate about poverty.

Closed data

All these ideas lead to the challenge of closed or private data. Open data is one driver to hope for more access, but maybe more could be done under the umbrella of corporate social responsibility or through the concept of data philanthropy. But is there any non-social media company offering a real-time API yet? There is need for precise data requirements, solutions for data portability and data protection. Especially mobile phone data with movement logs of its users contains highly sensitive data and cannot, as different studies show, be fully anonymized. This is the great side-effect about such personal data. How can it really be protected?

Many participants shared the hope that corporate data philanthropy would become the norm soon and perhaps a requirement to bid successfully on contracts. There was also a discussion on the fact that development organizations needed to make a better case for why corporations should share data with them. The open data movement has focused so far on governments, so maybe a shift to companies could help here.

Crowdsource data

Another stream of discussion went around data collection or producing data in need, since most data is not at hand and is more or less closed data. Could crowdsourcing approaches help here to get accurate data about poverty? There was an interesting discussion on personal data philanthropy and the growing willingness of individuals to 'donate' personal data for the public good) (e.g. quantifiedself.com). Efforts for crowdsourcing in recent years, show that user reported data can be an alternative to traditional surveys. Can these approaches be scaled to the level needed to get accurate results? The development of low-cost sensors is widening the landscape of data tracking. Looking at the growth of smart phone usage in developing countries, this is a potential venue for data collection in the near future. There have already been experiments to use sensors to control water access or to analyze the water quality across a country.

This out-of-the-box thinking shows new potentials for using data to help understand complex problems better. But it also became clear how little data is available in the public space; and to fight poverty, such data analysis can be only a tool to then act on. And the second step is far bigger and even more complex.