"A Toolkit for Evaluating Public R&D Investment: Models, Methods, and Findings from ATP's First Decade"

By Rosalie Ruegg and Irwin Feller, Advanced Technology Program, National Institute of Standards and Technology, July 2003.

Abstract: "Evaluation is an essential component of publicly funded R&D programs, both in support of program management and public policy. The Advanced Technology Program (ATP) has emerged over its first decade as a leader in evaluation, engaging nationally prominent evaluators to apply new and existing methods in building an analytical and empirical basis for ATP’s operations and performance.

This report draws from a body of 45 studies commissioned by ATP between 1990 and 2000 and analyzes the methods and techniques used and examines the findings of those studies. These studies have increased understanding not only of ATP but also of the dynamics of innovation systems and the relationships between public and private sector funding of R&D. The findings examined are organized around five major themes: firm/industry effects, collaboration effects, spillover effects, interfaces and comparisons with other programs, and measures of overall program performance.

The extensive toolkit of evaluation methods presented in the report illustrates how those methods can be used to answer a variety of stakeholder questions. Methods include survey, descriptive and economic case study, bibliometrics, historical tracing, econometrics, expert judgment, social network analysis, cost index, and a composite performance rating system constructed from indicator metrics. Additionally, the use of analytical and conceptual modeling to explore a program’s underlying relationships and process dynamics is considered. The political economy of ATP is discussed, and an evaluation framework and an overview of evaluation best practices are provided.

The report integrates and condenses a large body of related research and thus provides ATP with a convenient reference work, toolkit, and planning guide. For those administrators of other programs, public policy makers, and evaluators, the report also serves as an evaluation toolkit by providing a logical framework for program evaluation, illustrating the use of evaluation methods and techniques, providing an overview of evaluation principles and practices, organizing a body of knowledge on how public-private partnership programs function, and contributing to an understanding of what evaluation is and how it is practiced in the field of R&D."