Ronald Inglehart and other members of the World Values Survey executive committee visit the United Nations on Monday. (Pippa Norris)
On Monday, there was a meeting of at the United Nations headquarters in New York on the use of social surveys for monitoring the Sustainable Development Goals. The event was organized by International IDEA, the World Values Survey (WVS) and the United Nations Development Program. The well-attended meeting saw presentations from many members of the WVS executive committee, as well as by many experts in the international community.
The event was timely, as this year sees the official end of the Millennium Development Goals (MDGs), which the world’s governments agreed to in the United Nations at the beginning of the 21st century as the heart of the development agenda. The eight goals are monitored through 21 targets and 60 indicators. The 2014 MDG report assessing progress in achieving the goals suggest a mixed bag: Some targets have been met, with progress since 1990 in child survival, literacy and access to basic sanitation. Still, profound social disparities exist; so too does extreme poverty.
In the “End of Poverty,” proponents like Jeffrey Sachs press the case that technical know-how and learning can and has designed highly effective aid programs that save lives and strengthen development. By contrast, in “The White Man’s Burden,” skeptics such as William Easterly claim that most large-scale aid projects are doomed to fail. Reductions in global poverty can also be attributed primarily to the remarkable economic growth improving the lives and security of millions living in China, rather than development aid per se. Most debate about aid effectiveness has focused on examining tangible local results attributable to specific projects, such as initiatives promoting girl’s schooling in Afghanistan, child immunizations against measles in Nepal, or the distribution of anti-malarial drugs and insecticide treated bed-nets in South Sudan. An extensive literature has sought to determine the broader impact of aid using national indicators, such as trends in poverty, child mortality, primary school completion rates and the proportion of the population with access to clean water.
As the era of the MDGs draws to an end, the international community is debating their replacement in 2016 by the Sustainable Development Goals (SDGs). It is proposed to expand the number of goals, and also the statistical indicators and specific targets adopted to monitor progress. The plan has been criticized by the Economist as a ‘Christmas tree’ where there are so many indicators that monitoring will be a nightmare, overwhelming the capacity of national statistical offices to generate reliable data in many developing countries.
Statistics on many of the standard indicators used by the MDGs are incomplete, even concerning basic matters such as conventional measures of income poverty assessed by the proportion of the population living below a dollar a day. There is often a substantial time-lag between data collection and policy analysis needs. Moreover poverty and human development nowadays are increasingly understood as multidimensional phenomena where household access to cash income provides a poor proxy indicator of social deprivation, such as access to essential medicines, feelings of neighborhood security or experience of lived poverty. Official agencies in many fragile states with displaced populations and least developed economies have limited or no access to reliable decennial Census data, Labor Force or multi-topic Household Surveys providing estimates of multidimensional aspects of poverty. A recent report by the United Nations Inter Agency and Expert Group suggests that there remain many cross-national inconsistencies in harmonizing the definitions, sources, time-periods, and methods used to estimate progress towards the MDGs.
In response, several U.N. bodies have suggested that the international community should supplement official statistics by incorporating various innovative data sources associated with the ‘Big Data’ revolution. Hence the report, “A World that Counts,” recommends: “Better data and statistics will help governments track progress and make sure their decisions are evidence-based; they can also strengthen accountability. This is not just about governments. International agencies, CSOs and the private sector should be involved. A true data revolution would draw on existing and new sources of data to fully integrate statistics into decision making, promote open access to, and use of, data and ensure increased support for statistical systems.”
A lot of the proposed indicators for the SDGs are currently proposed to be measured using government provided official statistics derived from official records (such as birth registries and census data) and general household surveys, such as enrollment rates for girls and boys. Governments often prefer to rely upon official statistics, in part because they thereby control the production of data, and they can also ensure its quality and legitimacy. Yet in many countries of the world, even basic demographic information about births, marriage and deaths are not systematically collected; for example, 62 percent of developing countries have no reliable birth register. Where available, data collected by official statistical agencies also takes three to four years to be standardized by international bureaus before it can be disseminated for consistent cross-national comparisons, such as use by the World Development Indicators. And the production of even so-called ‘hard’ data, such as unemployment rates or estimates of gross domestic product, involve many technical and politically-sensitive decisions; for example the measurement of the unemployment rate in Britain was adjusted 19 times during the years under the Thatcher government. There are several forms of data which could be collected through Big Data – each important but slightly different. Official statistics are increasingly being supplemented by other forms of non-governmental data from a wide range of sources, of varying quality and coverage.
As one important aspect of this data revolution, many of the long-established cross-national social surveys can play a vital role by generating robust and reliable data useful for monitoring the SDGs. This includes studies such as the World Values Survey, founded in 1981 and now covering around 100 societies, and thus the grand-daddy of comparative social and attitudinal surveys by non-profit international organizations. It can also engage the Global-barometers covering several major world regions, such as the Afro-Barometer, and commercial surveys, such as the Gallup World Poll. Social surveys tap different forms of data. Some is experiential – such as the Lived Poverty index asking how much the respondent and their family had gone without various basic necessities of life during the previous 12 months, such as enough food to eat, necessary medical treatment or a cash income.
Another measure of personal experience is whether a person reports being a victim of a crime. Other measures are more perceptual, for example how far respondent feel unsafe from crime in their neighborhood, or whether they believe that the threat of terrorism has got worse. Both are essential to monitor. Experience of poverty has to be measured by asking ordinary people about their lives and household cash income is only a proxy for human security. And perceptual data is also essential, since this forms the social reality of what ordinary people believe. Ideally both experiential and perceptual data can be compared with other official statistics, to triangulate and thereby provide a more comprehensive, valid and legitimate picture of the world for policymakers.
There are many advantages for the international community in tapping into reliable and well-established social surveys involving a representative sample of ordinary people living in each society. One is that these can furnishing data on multidimensional experiences of lived poverty, such as self-reported access to clean water, to food security, and to medicine. Household access to cash income (such as living below a dollar a day) is an inadequate proxy for human development, especially in rural economies and exchange markets. Survey data can thereby enrich our understanding of the different types of severe challenges commonly facing poor households.
Surveys are also well-designed to measure public perceptions of ordinary people, which is essential to monitor subjective feelings of security, attitudes towards social deprivation or satisfaction with public services. In addition, reliable social survey data can also be disaggregated to examine inequalities among major social sectors, such as between women and men, young and old, as well as rural and low-income households. Concern about growing inequality within nations, even while GDP has risen, has pushed concern about this issue to the forefront of the development agenda.
At the same time official national statistical offices may be wary about using social surveys for several reasons. The size of the national samples used in most standard social surveys is far more limited than in official Household Surveys or population estimates based on the official national census. Nevertheless the sample size of 1,000 or more people in each society, used in social surveys, is widely accepted as standard in many public opinion polls and it is appropriate so long as the sampling method and fieldwork procedures are well-designed and the results are published along with transparent technical information about confidence intervals.
Surveys also need to be careful in harmonizing demographic and social characteristics, so that standardized procedures are implemented across diverse societies. Moreover data should be freely disseminated in user-friendly format without cost as a public resource for the international community, with transparent technical details about sampling methods and questionnaire design, so that national statistical offices, local experts, NGOs, and scholars can access, scrutinize and analyze the data to reflect local priorities. By expanding the use of social survey resources in developing countries, this also strengthens local market research and statistical capacities, as well as potentially providing a voice for ordinary people in shaping development priorities.
Click to enlarge
How could this data be used? Table 1 illustrates the potential for generating a Development Dashboard and applying the results from selected items contained in the sixth wave of the World Values Survey (WVS-6) as benchmarks to monitor progress towards the Sustainable Development Goals. WVS-6 covers sixty societies and the case of Ghana was chosen to illustrate some of the results. Thus for example the first SDG is to end poverty. To monitor this, the WVS asks: “In the last 12 months, how often have you or your family…Gone without a cash income?” While 1 in 10 Ghanaians reported going without a cash income in 2012, but the number rose to one fifth of the older population and one quarter of the lowest income households. Similarly, to monitor food security, the WVS asks: “How often you or your family have gone without enough food to eat in the last 12 months?” While one fifth of people living in Ghana reported going without food, the proportion rose to 30 percent among the older population and 43 percent of lowest income households. Similar social disparities can be observed across the range of indicators. Moreover concern about subjective security risks were far higher than the reported experiential risks in Ghana, for example there are widespread worries about losing a job or not being able to give children a good quality education. The Development Dashboard gives all actors — bilateral donors, local policymakers, civil society monitoring organizations and ordinary citizens — the capacity to see transparently how far developmental goals and targets are being met – and where we are falling short of our aspirations.
Thus post-Rio, the world is facing multiple challenges in meeting developmental goals. One of the lessons from the MDGs is that setting specific goals and concrete targets can be a useful stimulus to focus attention on several critical problems facing the world’s poorest societies, encouraging the delivery of results and thereby holding governments to account for their investment of development aid. But the results-based approach is data intensive. Better monitoring requires taking advantage of the leaps in data availability which have become widely available through the information revolution in recent decades, including drawing upon social surveys as one important source of evidence in the international community toolkit.
Pippa Norris is the McGuire lecturer in comparative politics at the John F. Kennedy School of Government at Harvard University and ARC laureate fellow and professor of government and international relations at the University of Sydney.