Judging the success of a technological innovation

The standard cost-effectiveness, cost-efficiency, engineering economics approach to evaluating the success or failure of a project are well known, and this is not the place to review those methods.

I want rather to make some different points to show why so much of the literature is wrong. Thus in spite of the fact that information and communications technology have swept the world at an unprecedented rate, and that few Americans or American businesses would consider a future without their installed technology, there is a wide spread belief that most ICT projects fail.

Most of the literature seems to consider "stand alone" projects that have specific objectives, specific budgets and a schedule. Thus projects are judged to have failed if the specified objectives are not met within the specified budget and schedule. A more reasonable criterion would be whether the project achieved enough to justify the effort involved and the resources it used.

There is an alternative literature that considers the process by which technological improvement takes place. In that literature, the process can be considered a success even if some of the specific innovation projects along the way don't meet their nominal objectives. Indeed, that literature suggests that projects be sequenced in such a way as to best influence the hearts and minds of the people in the organization in question. The first projects in such a process might be chosen for their simplicity, visibility and probable positive impact on people's attitudes. Indeed, I suggest that the best strategy for the improvement of technology in an organization might in some cases involve the deliberate programming of sub-projects with unrealistic objectives, budgets and calendars.

One of the problems with the project by project evaluations of an innovation process is that early projects often have externalities that affect the overall process, but are difficult to measure. Thus, an early project while not meeting its explicit objectives might well leave a residue of skills in organizational personnel, of physical infrastructure that will have unplanned uses in the future, and of improved understanding of the organization and its processes in the change agent team. None of these would be likely to be measured in evaluating the project per se.

The rapid dissemination of ICT technology in the United States has been described as a viral rather than a planned process. Huge numbers of individuals and individual firms made decisions apparently independently to install technology and learn to use it. There was no centrally planned process that lead to the rapid dissemination of the technology nor to its eventual improvement of economic efficiency of the country. Yet many donor agencies now focus on scale up of pilot projects as the means of enhancing dissemination rather than creating conditions propitious to individual initiative. The question that should be asked of such efforts is whether they were more or less effective in promoting the dissemination of the technology and its effective use than other available approaches, not whether they achieved their stated objectives under budget and in time.

It has been recognized that the economics of network technologies are unusual. The benefits tend to accrue according to the number of participants in the network squared while the costs increase according to the number of participants alone. Since networks start small and grow, the benefit to cost ratio early in network development tends to be much lower than the ratio when the network reaches maturity. Unfortunately, many project evaluations are done over relatively short times, early in the development of the networks.

As we look at the development of personal computers and the Internet, we recognize that killer application succeeded killer application, and that today's value of the information infrastructure could not have been predicted, because no one had the foresight needed to predict those killer apps nor their importance and value. So too, as an organization begins the process of technological innovation it is unlikely to predict where it will lead nor the benefits it will yield. If the early projects are steps toward that uncertain future, how then can they be evaluated?