This talk by Stacey Young is from the KM Impact Challenge of the Knowledge-Driven International Development Project of USAID. Dr. Young makes a great point that many factors effect the success of a development project, and in the complex world of developing nations it is likely that the combination of factors that lead to success in one place will not exist in the next place. She doubts the utility of cookie-cutter approaches, preferring rather an approach in which there is careful monitoring of each project, always informed by knowledge, leading to modification of designs and objectives according to the local situation to achieve success. She notes that the things that she thinks can be useful to measure how good a knowledge management project is are seldom quantifiable, while donor agencies these days tend to be slaves to quantitative evaluation.
I wonder how good the evaluation has been of evaluation efforts. What are the "quantifiable" indicators of the information that evaluations provide decision makers? What is the ratio of the improved yield of donor program expenditures due to evaluation efforts to the costs of those efforts? Of course, there is no satisfactory information of this kind, in part because monitoring and evaluation are knowledge activities that are by nature poorly suited for such quantitative approaches.
What are other knowledge based programs, and how are they evaluated. Teaching is such a knowledge-based program. In my teaching I have found course evaluations by students to be helpful, but I suspect that the university goes more by whether students sign up for the courses each semester in sufficient numbers that the course costs are adequately paid. I note that business schools use the average salaries of the graduates on entering the job market as an indicator (which would seem likely to be of use in attracting future students). I note that billionaires Bill Gates, Steve Jobs and Mark Zuckerberg left college early and probably had rather low starting incomes as they began to make their billions; perhaps starting salaries on graduation are not the best indicator of social value!
Research is another knowledge based program. I recall that the National Science Foundation found in a study of research projects that had made major discoveries, a significant portion of those discoveries were not mentioned as possible outcomes in project proposals. This is an illustration of the fact that decision on research are made with incomplete information and limited rationality. Still, I suspect the process of planning and evaluating scientific research projects typical of those funded by the National Science Foundation is more informed and more rational than that of assistance projects funded by NGO's or donot agencies.
I have long been concerned about the conceptualization of small projects, and the use of terms such as "pilot projects" and "demonstration projects"; terms such as "going to scale" and "scaling up" also concern me. The diffusion of innovations is often more of a viral than bureaucratic process, and if done well involves innovation, deepening, and tailoring at each level. It may be that one can test a curriculum with a group of teachers, but I bet that when that curriculum is rolled out to a wider user group each good teacher modifies it to meet the needs of her students and to fit the style and approach to teaching he/she finds to work best. In the small projects funded by the programs that I ran, knowledge was generated and it was diffused by publication, by teaching, by example, and by other means, influencing others in many ways.
Perhaps "monitoring" is a better term than "evaluation". Perhaps the effort should be based on finding out what is actually happening, rather than trying to measure how fully the original conceptions (made with incomplete information and limited rationality) are actually fulfilled.
Not everything that can be measured is worth measuring.
We don't know how to measure everything that is important.
Good decisions are sometimes made on unsatisfactory information and analysis.
Bad decisions are often made on the basis of pretty good information and analysis.
Development seems to me something to tackle with a fair amount of humility and a willingness to reexamine assumptions and revise plans on a continuing basis. I do like to quantify and I do like to ask experts for their opinion and I do like to analyse that information that I can obtain, but I try not to take myself too seriously!
No comments:
Post a Comment