A thought about quantitative evidence

Evaluation professionals seem to emphasize quantitative evidence over qualitative. It is more important to get evidence that contributes to the kind of understanding that will lead to decisions that improve the situation, than to get evidence that satisfies the aesthetics of evaluation professionals.

When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.
Sir William Thomson, Lord Kelvin

Not everything that can be counted counts and not everything that counts can be counted 
William Bruce Cameron

The problem is, of course, that many people seem to utilize quantitative data in their analysis whether or not it counts. Your knowledge may be of a meagre and unsatisfactory kind, but if it is all you have and it applies to the problem at hand, use it!

I am the manager of the Monitoring and Evaluation group on Zunia. I am also a member of the Monitoring and Evaluation Professionals group on Linked In. In the contexts of those groups I have been worrying about the framing of questions.

Let me give an example. I recall a colleague in a developing country telling me that he had been having difficulty understanding why health indicators were so poor in one region. He had come to realize that public health services were very deficient in that region, but that only led to a further question. Why were public health services not focused on a region with evident public health problems. It was only when he recognized that the region was the stronghold of the opposition to the party in power in the central government that illumination dawned. The problem was not one of health services, but a political problem -- the government policy under the current leadership was one of retribution against its political opponents. The government was doing everything possible to make the people of the region suffer economically, politically, and even with poor health.

If one had tried to "evaluate" the public health service and its "impact." without understanding the political and economic situation, one would have been gravely misled. One might have recommended strengthening the public health service budget and staff (and dealing with poor morale of service providers) in the assumption that a stronger health service would improve health status. Recommendations of that nature would almost surely not have been implemented; if implemented they might have had little impact on health status which depends on nutrition and water and sanitation; with high levels of unemployment, high levels of poverty, and poor governmental services health status might not improve even with improved health services.

The point is that it is important to ask the right questions if you want the answers to lead to real development. Generally the way to be sure to ask the right questions is to ask a lot of questions in the hope that at least some will pay off. Asking questions that only quantitative evidence can answer is probably not sufficient to really understand the situation. Of course, where quantitative evidence is available, it can often be very useful when taken in context.