Dan Ariely: Beware conflicts of interest

Another good talk from Dan Ariely, a psychologist who can I think really help you to think better and more effectively.

In this talk he describes an experiment that failed to reach statistical significance, an effect that he associated with a drunk coming into the experimental group that was supposed to do well and by doing very badly dropped the average performance for the whole group a great deal. He draws from this experience the conclusion that one should report fairly what one actually observed and not throw out the outliers. It is hard to quarrel with that statement.

I suggest however that there is another lesson. In planning an experiment with human subjects, specify carefully the qualifications that a subject must have to be allowed into the experiment. Don't allow a drunk into an experiment testing intellectual competence. Indeed, balance your experimental group. For example, you might pair subjects to be as similar as possible on all the relevant characteristics and then assign one member of each pair randomly to either the experimental or the control group, assigning the other subject to the other group.

Ariely recommends not biasing your results by throwing out observations you don't like.

I recommend also that you plan your experiment carefully and work to assure that your groups are really comparable on all but the experimental treatment.