Measures that matter -- for international comparisons

I quote from an article in The Economist (which is also the source of the above chart):

(P)erformance indices, which rank social issues or policy outcomes in different countries by combining related measures into a single score for each, are enjoying a boom. Their number has soared over the past two decades (see chart). For many issues, rival indices must now battle it out...... 

The best indices are meticulous (PISA, for instance, combines dozens of carefully standardised sub-measures and raises statistical caveats). But others are based on shaky figures that are calculated differently in different countries. And choosing what to include often means pinning down slippery concepts and making subjective judgments. An index of democracy, freedom or happiness means putting hard numbers to the fairness of elections, weighing civil liberties against economic rights, or deciding how much to rely on surveys.

The article goes on to discuss work by Judith Kelley of Duke University and Beth Simmons of Harvard University on the Trafficking in Persons (TIP) index "first published in 2001. That year’s annual report covered 79 countries; it now ranks almost 190." The publication of the index data is described as having influenced countries to pass laws against trafficking, but the report as suffering from difficulty in accurately measuring the numbers of persons trafficked.

Of course, the indicators and reports are not of uniform quality. Some, such as the OECD’s Program for International Student Assessment (PISA), which rates 15-year-olds’ academic performance in dozens of countries, are very good indeed. Others require a fairly high degree of sophistication to properly understand and utilize.

Indicators are hard work! It is hard to choose an indicator that is measurable, for which the measurements will be reasonably accurate, that will be policy relevant, and that will be reasonably well understood by the policy makers who are to use the data.

• GDP, for example, leaves out the product of unpaid services (such as those done by family members in the home) and can be very inaccurate in measuring the product of the informal economy, and even less so in measuring the product of illegal activity.
• Employment is difficult to interpret as one deals with people who want jobs but are too discouraged to actively seek them, people (such as retirees) who would work if approached with the right jobs but who don't respond that they want work, people who work part time (especially in more than one job), and people who work in the informal economy, in subsistence agriculture, and in the illegal economy.
• Health: while WHO has promoted the use of disability adjusted life years (DALYs), I don't think it fully captures the burden of illness, nor is fully useful as an instrument for policy makers.
• Happiness has been experimented by some countries and researchers, but has obvious problems, especially in comparing one cultural group against another (say a phlegmatic group versus a vocally complaining group).

One wants indicators to be used consistently over time so that policy makers can see the overall trends and understand the impacts of their policies; one also wants to update indicators to take advantage of improved theory, understanding, and data collection methods. Unfortunately, the two objectives are mutually incompatible.

Still, I think the availability of more indicators and more reports giving international comparisons based on the best available data is likely to be a healthy trend. After all, there are different audiences for the different reports, and we all like to do well as compared with our peers.