Monday, November 30, 2009

"Why feedback can be just so much noise"

My friend Julianne also recommends this article by Tim Harford suggesting that feedback is not as great as many naive people assume. In theory we should be monitoring and evaluating our programs and projects, adjusting them to improve performance in accord with user feedback. In practice, M&E is often confusing and even misleading, in part because the questions asked of users are often based on poorly understood models of reality or because the questions are badly designed to measure what they are intended to reveal.

The article draws on Streetlights and Shadows: Searching for the Keys to Adaptive Decision Making by Gary Klein. I haven't read the book, but the reviews look good! I quote from a review of the book:
Klein offers more realistic ideas about how to make decisions in real-life settings. He provides many examples—ranging from airline pilots and weather forecasters to sports announcers and Captain Jack Aubrey in Patrick O'Brian's Master and Commander novels—to make his point. All these decision makers saw things that others didn't. They used their expertise to pick up cues and to discern patterns and trends. We can make better decisions, Klein tells us, if we are prepared for complexity and ambiguity and if we will stop expecting the data to tell us everything.
In researching this posting, I came across this great site from the Air War College on thinking and decision making, full of links to practical guides.

No comments: