Monitoring and Evaluation: ICT

These are not us chatting!

I was chatting with a graduate student about monitoring and evaluation of efforts to disseminate ICT to developing nations, and it occurred to me to post some of my thoughts on the topic.

Planned Versus Viral Dissemination

The first major distinction I would make is probably between planned and viral processes for the dissemination of technology. It seems often my more bureaucratic friends think about pilot projects which, when tinkered into success, are scaled up. Their planned, deterministic approach might be applied to the introduction of ICTs in the public medical care system, the public school system, or the development of a national system of telecenters in rural areas. It is similar to the approach for innovation in an organization which creates a center for the innovation followed by a process of dissemination to other organizational units.

The viral approach, easy for me as an American to identify with, involves setting up the conditions that encourage people to innovate and creating the institutions needed for them to obtain the resources with which to innovate, and letting inventions diffuse. People observe their neighbors successes and failures, obtain information from all sorts of sources, and adapt the technology as best they can to their own needs and desires. The process is unplanned, teleonomic rather than telologic, but is the preferred approach in our U.S. post-frontier, anti-authoritarian culture.

Both approaches have their place, and indeed they may be complementary. But for the purposes of monitoring and evaluation, the two approaches have differing meanings.

I suggest, however, that projects should always be evaluated, and seldom are so evaluated, in terms of their contribution to the overall development of the ICT infrastructure and the appropriate evolution of the institutions utilizing that infrastructure. Are the projects properly chosen for the state of the infrastructure and institution? Could the project resources have been better used to grow the infrastructure faster or better, to make the institution function better or to achieve institutional improvement more quickly?

Projects Versus Infrastructure and Institution Building

I would also make a distinction between projects and the longer range process of change of the ICT infrastructure and change of the institutions built upon that infrastructure. Projects are best understood within that larger context.


The information revolution has been going on in its pioneer societies for a very long time. Computers were first introduced in World War II, and telephones in the last quarter of the 19th century; the telegraph even earlier. Large organizations in developed nations not only have a long experience of building their ICT infrastructure, they function in ways made possible only by that infrastructure. So too, modern nations have long histories of building national information and communications infrastructures and of elaboration of changes in social and economic institutions to adapt to the challenges and opportunities of their evolving infrastructures.

It is platitudinous to say that most ICT projects fail, but it is also obvious that institutions have changed enormously due to huge investments in ICT and are vastly more efficient and effective as a result of the information technology revolution. I think the answer is that there have been unrecognized positive externalities of these failed projects. And of course that the viral processes that have been going on in parallel with the planned projects have also yielded great benefits.

With the foregoing as preamble, let me suggest that the terms we use to describe "projects" matter. Here are some common ones:

• A "pilot project" is a project conceived of as developing and testing ICT approaches that, if successful at the pilot stage, would be more widely applied;
• A "demonstration project" is one conceived of as demonstrating ICT approaches known to work well in one or more situations to people who might adopt them in other situations;
• A "research and development project" is one that is conceived as seeking to develop new ICT approaches, leaving their diffusion to later efforts;
• We might also consider a "generational project" as one which develops the next generation of an ICT approach. Thus chip manufactures develop generations of chips, making massive changes in design and manufacturing processes for each generation; Microsoft replaces DOS with Windows, and then Windows with Vista.

On the one hand, it is important in monitoring and evaluation of a project to be specific as to the objectives of that project. On the other hand, it is also important to recognize that a projects effects can be seen and evaluated in the larger context.

I would caution against mis-designation of projects. If indeed an ICT application is at the early stage best characterized as research and development, it is unfair to evaluate it as a pilot or demonstration project. I would also caution against allowing one's preconceptions of the nature of the project from blinding one to the actual results of the project. A pilot project may not be scaled up, but may instead be widely influential in the diffusion of the technology via other means; is it then a failure?

I had difficulties with this kind of error when I was the Work Program Director of the infoDev program. The projects we funded were intended to be innovative projects, introducing a new technology into their countries and often inventing new ICT approaches. A survey I did identified that the project personnel were publishing their findings, were teaching in universities building on their experience, were consulting, and were demonstrating their efforts to many (apparently) influential visitors. ICT officers of donor agencies were gaining experience with ICT applications via exposure to those projects, and were presumably utilizing that experience in other project development. We had no way to measure the impact of those projects on the actual dissemination of ICT in developing nations, but I continue to believe that there was such an impact.

On the other hand, few of the projects reported that their projects had been adopted by a government as pilots and had been scaled up. The program was criticized by its donors for the lack of that kind of impact. I think the issue was in the distance between the preconceptions of donor staff, infoDev staff and project staffs. However, it seems to me that without means of measuring the actual impact of the portfolio, it was unfairly criticized as failing to meet preconceptions of a few donor agency staff members.



Project Portfolios

Portfolios of projects raise their own monitoring and evaluation issues. Often people suggest that each project in a portfolio devote a given percentage of its budget to monitoring and evaluation. This can not be right! Consider the total resource base available for monitoring and evaluation of the portfolio. The field of possible alternatives for the allocation of these resources among the projects of the portfolio is infinite. The probability approaches zero therefore that the optimum allocation is either equal amounts to each project or the proportional allocation according to project budgets. I suggest that a heuristic allocation based on the issues the portfolio managers are most willing and able to address is likely to be better than any nominal allocation. (Incidentally, the limiting resource may not be funding, but rather things like human resources or the time of key decision makers.)

I managed portfolios of projects for many years and I found some useful approaches to be a combination of:

• Ex anti peer review of project proposals, in a multistep process in which more time and effort are devoted to the proposals most likely to merit funding (with feedback to project proponents);
• Regular reporting on project progress from all projects (but not so often as to constitute an excessive burden to the project personnel not to overwhelm the staff reviewing the progress reports);
• Portfolio management staff review and feedback on all project reports;
• Focus groups of representative project implementers with portfolio management staff for a combination of open discussion and directed discussion of key issues;
• Ad hoc ex post peer reviews of groups of projects, where the issues to be addressed are identified in advance and the projects are selected as those best suited for the purpose of addressing those issues.
  

The Problem of Intentionality

As described above, infoDev was an intermediary between large donor agencies and small grant recipients. (Large donors generally don't make small grants, since their management often believe their agencies can not do so efficiently and effectively.) The result was that the program was held by each of its donors responsible for achieving that donor agencies objectives in funding infoDev. The staff of infoDev, which came from a number of backgrounds, had their own intentions for the program. Of course, the grantees all had their own objectives, and most sought only an acceptable overlap with the intentions that they perceived (not always accurately) of infoDev.

This is an example of the general problem that different stakeholders in the application of ICT to development have different intentions. My experience is that often the intentions are not articulated, and perhaps not even fully appreciated by those involved. Successful writers of proposals are expert at hiding their own intentions and explaining how the proposed effort is likely to meet the perceived objectives of the funding agency. In short, efforts to evaluate whether efforts have achieved objectives run into the problem of determining what the objectives were, and whose objectives count for how much.

Even dealing with projects prepared by skilled international bureaucrats in organizations with strong bureaucratic procedures to assure clear statements of objectives, I have found the stated objectives of projects to be hard to understand, and have found different interpretations of those stated objectives by different stakeholders.

An alternative to the effort to evaluate the fulfillment of project objectives is an effort to document project effects, both positive and negative, and to compare those with project costs. The evaluator can then come to his/her own conclusion as to whether the benefits outweigh the costs, recalling that effects that can not be measured or quantified may still exist.

Which brings me to the unintended consequences of projects. I am reminded of the creators of a network of telecenters in villages in India who discovered to their surprise that its most important application in the minds of the villagers appeared to be in simplifying the search for suitable marriage partners for their young people from other, nearby villages. The organizers had assumed that the benefits would come from agriculture, public health and other "serious" applications. How foolish it would be to denigrate the advantages that were recognized by the people themselves. In a similar vein, the telephone companies originally tried to dissuade the household use of telephones, feeling the network should be reserved for "important" uses in business. Consumer demand eventually overcame that foolish bias.

ICT As a Means to an End: The Matrix of Objectives

The Logical Framework is a well known approach to the management of development projects, first developed and disseminated by Leon Rosenberg and my friends Larry Posner and Molly Hageboeck. Using the LogFrame, projects are described in terms of their inputs, outputs, immediate purposes and longer term objectives. It encourages assumptions to be documented, and requires the project personnel to define specific indices and benchmarks against which project performance is to be measured. The approach can be adopted for monitoring and evaluation of projects introducing ICT applications, and indeed has often been so adopted.

Let me suggest, however, that it might be more appropriate to consider ICT applications more specifically using a framework for the sector in which the technology is to be applied. Consider for example, health services. We may consider health services to be divided into several categories according to the institutions which provide them:

• health education and communications delivered via mass media, in schools, etc.;
• public health services such as immunizations delivered house to house or otherwise in the community;
• outpatient medical services delivered via health centers, emergency rooms, physicians offices, etc.;
• inpatient medical services delivered via hospitals of various kinds.

We can also consider a hierarchy of measures that can be applied to the evaluation of health services, such as:

• benefit to cost ratio
• overall cost
• effectiveness (measured over the population)
• coverage
• efficacy (measured in the individuals receiving interventions)
• quantity of service output
• quality of service output
• cost per unit service output
• financial inputs
• quantity and quality of human resource inputs
• quality of the technology (body of techniques utilized)
• quality and magnitude of the physical plant used to deliver services
• quality of the management of the service

I consider ICT in this context to include a wide variety of technologies, and indeed it may be that formal review of the possibilities of application of the variety of technologies to the variety of functions of the health service will lead to interesting and useful innovations. Consider:

• broadcast radio and television, including community radio;
• recorded media such as cassette recordings, VCR,
  
• citizens band, short wave and other point to point to radio
  
• mobile and fixed line telephone.
• call center technologies
  
• PDAs and other hand held computers,
• personal computers
• computer centers
• networks, local area networks, wide area networks, Internet
• imaging technologies and other ICT embedded in diagnostic or other medical devices
  

Such a broad framework encourages one to consider whether the resources available for introducing and disseminating ICT innovations in the health sector are being used most effectively to improve the health of the target population.

For example, it encourages one to ask:

• Whether innovations in communications should best be used to influence the general public via broadcast media, or target populations of patients via small group or individual sessions, or health service providers?
• Whether telemedicine should be introduced focusing on service to the homebound patient, the community health auxiliary, backup of primary care physicians with specialists, etc.?
• Whether computer support should prioritize applications in service management or in the delivery of care?

Returning to earlier themes, the health system as broadly conceived has obviously been evolving its ICT infrastructure for decades and has been evolving institutional responses to the changing infrastructure over a similar period. I find myself communicating with my HMO via the Internet in ways I could not have imagined thirty or forty years ago. On a visit to my physician, I am greeted by a nurse with a fully automated set of instruments that communicate directly with ah HMO wide network. All the providers in the system face me with computer terminals at their sides, entering and retrieving information from my computerized medical records. I also have had medical imaging via computerized tomography. The examples go on and on of service involving ICT that I could not have imagined in the past.

Thus the introduction of ICT in the health system must necessarily be considered a very long term process which, especially in developing nations, is at least decades from completion. It is important to manage that long term process so as to utilize scarce resources as effectively as possible, ultimately to maximize the improvements in the health of the population being served.

Doing so must necessarily involve projectizing interventions to improve the ICT infrastructure and its applications. Thus there is a need for project monitoring and evaluation, as well as the longer term monitoring and evaluation of the development of the ICT infrastructure and its applications for health services. Indeed, in my HMO I am sure that at any given time there is an ongoing portfolio of ICT related projects being simultaneously implemented.

In the process, the interests of many stakeholders have to be considered. Patients are not an undifferentiated collective these days, but may be organized in community or diagnostic categories. The interests of physicians, nurses, pharmacists, and other professionals in a health service are not identical and may be conflicting. The public has interests both as those paying for the health service system and as the patients of that system, and the guardians of these interests may be different one from the other.

You can find many resources on monitoring and evaluation of ICT projects on the Development Gateway's site, which at the moment is not available to accept new postings.