A thought about reading people -- a measure of credibility

Two things came together in my mind, leading to this posting.

• Discussions with graduate students on methodologies for their class research projects;
• Reading The Tailor and Ansty by Eric Cross.

The latter is a book about an elderly couple, living in the west of Ireland in the late 1930's. It is full of recordings of talk by the tailor, who was a great talker. (The book, which was in print for decades, and may still be for all I know, gives a picture of Ireland that I found very interesting; it is a picture of a very traditional society, even at the outset of World War II.)

Now my father was born and brought up in Ireland. If you asked him a question, like the tailor he would generally answer. At the end of his answer, he might be convinced of his accuracy, at least for a while, even when he was winging it at the start. While a very honest man, he might give an answer designed to please the narrator by its inventiveness, its humor, or simply for the pleasure of confounding the listener.

Interviews with key informants are an important research method. Similar to my father, a respondent when asked a question may or may not tell the questioner what he believes to be the correct answer, and the respondent may or may not know the answer. Thus we have the possibilities:

• respondent tells what he believes, and knows the answer
• respondent tells what he believes, and doesn't know the answer
  
• respondent doesn't tell what he believes, and knows the answer
• respondent doesn't tell what he believes, and doesn't know the answer.

One could estimate the probability of each of these situations in advance of the question. One ought to have an idea of whether the person knows the answer to a question before it is asked, based on general knowledge of the person's relationship to the issue in play. So too, one might make a guesstimate as to the probabilities that the person will tell you what he believes given that he does, or does not know the answer. (not necessarily the same).

When you watch a person respond to a question, you might change your estimate as to whether the person knows or does not know the answer, and whether the person is being frank or not.

Information theorists define a quantity called entropy, considered as the average uncertainty, as the sum of the probability of a situation times the logarithm of its probability, summed over all possible situations. Thus in principle one could calculate the entropy over the set of probabilities of the four circumstances enumerated above. The entropy could be calculated both before and after interviewing a respondent. The difference between the entropy after the questioning and before is then the information obtained in the questioning.

Note, that in this case, one is measuring not the information gained about the issue in question, but about the credibility of the respondent.

Do researchers really use such an approach? If not, perhaps they should!

Of course, one could go further, and ask a follow-up question as to how sure the respondent was about the answer given. One would then have information to estimate the credibility of the respondent on the second question itself, and a broader basis on which to make the ex-post estimates of the probabilities relating to the initial question. (The credibility of the respondent can depend on the issue at hand -- one may know more about one issue than another; one may be more disposed to answer "truthfully" about one issue than another. A respondent may be more or less credible in the estimate of the accuracy of a response than in the response itself.)

I certainly hope that the students realize that the results of interviews with key informants should be treated as data, not fact. Informants are people. People don't always observe accurately or remember accurately what they observed. People don't always tell the truth, even as they see the truth.

The questioning process should also provide information on the issue behind the question. There should be (at least in principle) a priory estimates of different alternatives related to the issue, and a posteriori estimates, and the information obtained from the interview might be calculated as the change in entropy.

I am not sure how one might expect the information gained about the respondent to be related to the information gained about the issue addressed in the question. The situation might be complicated. Thus:

• If in the interview, one found that the respondent initially thought to be a good source turned out to be unreliable, then one would expect a lot of information about the respondent, and not much about the issue.
  
• Similarly, if both ex ante and ex post, the informant seemed a good source -- both knowledgeable and frank -- one might expect a lot of information about the issue, but little about the informant from the interview.

Perhaps one could formally combine the ex-post estimates of the probabilities each respondent was both honest and frank, with the ex post estimates of the different alternatives for the issue based on each respondent's answers, to make judgments on the set of interviews.