In this 4th in a series, the use of probability analysis to improve the information content of ratings and rankings is extended to ex post evaluation of projects as well as ex ante review. The approach could be used in a variety of development projects. It might even be used in the context of village decision making.I have recently added a series of posts on the use of probability theory, especially Bayes rule, and information theory to quantify peer reviewer information.
- Probability Analysis to Improve Scientific Peer Review
- Subjective Probabilities, Ratings, Handicappers, and Open Access Online Scientific Literature
- From Estimating Probabilities of Ratings to Figures of Merit and Rankings
While the first posts focused on scientific peer review, the approach could be used in ex ante review of any set of alternative proposals. It could also be used in ex post review of completed projects, that is in their ex post evaluation. Let me suggest a very different application to illustrate the point.
A Village Level Example
Think about a village in a developing nation in which a number of development projects are possible. It is desired to develop community participation in the ex ante selection of projects to implement, and to obtain community evaluations of the success of the projects when completed. In such villages it is sometimes difficult to get people to give their real opinions in public, where they may be different than the opinions of village chiefs, those of their family members, or of a perceived majority. It might be that a hole-in-the-wall computer would allow community voting in a less threatening situation. (The computer could of course carry out many other functions while not being used for voting and improving the interpretation of the votes.)
A hole-in-the-wall computer might be installed in the village, including a card reader that could be used to identify the current user. For example a plastic card like a credit card might be given to each adult in the village identifying that person. Perhaps a touch screen could be used to allow people to vote on projects, and the projects could be identified by photos (as well as text) to allow illiterate adults to vote. (Note that readable ID cards would also be useful for other applications, such as for health information provision.)
A meeting could be held with the community to build interest in and support for community development through community projects. In the meeting, the participants could learn to use the computer to vote on projects. In the learning process they could record some judgments on hypothetical projects; doing so could be done in such a way as to increase enthusiasm for the project based approach to development. It would also provide a basis of data for the process described in the above references.
People might for example give one to five stars to each project, such as is done in Amazon.com reviews of books. The projects could be ranked according to the average number of stars given by community members.
Once the initial meeting was complete, a list of projects could be programmed into the computer and the village invited to vote within a given period. Initially, the selection of the first projects to implement might be simply by summing the number of stars given for each project, divided by the number of voters participating.
When projects were completed, the community could be invited to rate the success of each, using the same scale for the voting. Indeed, one might have a vote at the completion of the project on how well the project went, and a later vote on how much the community liked the service provided to it as a result of the project.
When enough data had been built up on the rating performance of the community members, the approaches described in the previous posts could be used to refine the interpretation of the voting.
Scale Up of the Idea
A rural development program might include hundreds of villages, each of which might use the technique to improve decision making and to track the quality of its projects.
Consider instead of a village, using the approach in a city. Commissions such as land use planning commissions could be provided with software using the statistical approach that I have described for selection of projects. It might even be useful in something like a grand jury context.
Similarly one might identify a thousand citizens of a community as a standing committee to evaluate local government performance. At any time, a number of projects and services might be listed on a website for the standing committee, and the members invited to enter their evaluations. At start up of the effort (or affiliation of a new member), a number of past projects might be evaluated to obtain required information on member evaluation performance. Then results might then be used by government officials in their management, or published to promote open government.
A donor agency might fund a program to strengthen intermediate cities, perhaps to provide an alternative to excessive growth of the largest city in a country to to provide for alternative poles of development. The approach could be used in each of the secondary cities.
Other Applications for Improvement of Review Performance
There would appear to be many other applications. Some possibilities are:
Scale Up of the Idea
A rural development program might include hundreds of villages, each of which might use the technique to improve decision making and to track the quality of its projects.
Consider instead of a village, using the approach in a city. Commissions such as land use planning commissions could be provided with software using the statistical approach that I have described for selection of projects. It might even be useful in something like a grand jury context.
Similarly one might identify a thousand citizens of a community as a standing committee to evaluate local government performance. At any time, a number of projects and services might be listed on a website for the standing committee, and the members invited to enter their evaluations. At start up of the effort (or affiliation of a new member), a number of past projects might be evaluated to obtain required information on member evaluation performance. Then results might then be used by government officials in their management, or published to promote open government.
A donor agency might fund a program to strengthen intermediate cities, perhaps to provide an alternative to excessive growth of the largest city in a country to to provide for alternative poles of development. The approach could be used in each of the secondary cities.
Other Applications for Improvement of Review Performance
There would appear to be many other applications. Some possibilities are:
- Finance: Donor agencies provide support to banking institutions, and the decisions on individual loans by banks would seem a likely application for probabilistic approaches to improve choice and evaluate choices made. This may be especially true in microfinance projects.
- Technology innovation projects, in which large numbers of grants or loans are made to technological entrepreneurs, either to develop new technologies or to develop new small enterprises based on technological innovations.
- Education projects that use peer review of teacher performance.
- Health projects that use expert review of primary health worker performances. Peer review may also be introduced into hospitals where physician peer review is done of cases treated in the hospital.
Use your own imagination to construct other ways of voting on ex ante and ex post evaluation of alternatives using the probabilistic approach.
No comments:
Post a Comment