A thought about probability estimates

Apparently philosophers have stopped trying to demark a boundary between science and pseudo science. (Remember Newton devoted time to astrology and theology as well as to physics and mathematics.)

There are pseudo sciences in which peer reviewed journals are published. No one considers himself a pseudo scientist. Some scientists maintain that the use of experiments designed to contradict theory is the sine qua non of science, but many of the observational sciences (systematic biology, astronomy, geology, economics, sociology) are more about classification and observation than experiment. Maybe the problem is that "sciences" and "pseudo sciences" are found along an epistemological continuum.

Credibility and probability are concepts that are inherently based on concepts of continuums rather than binary concepts.

Truth on the other hand is a binary concept: something is true or it is not true. However, there is fuzzy logic which includes an intermediate truth value between "true" and "false".

I rather like the idea of probability on probability. Consider a situation in which you ask an expert handicapper what is the probability of a given horse winning a given race. Handicappers are good at such judgments.

Correlation of the subjective estimates of the probability a horse will win with the empirical frequency that the horse wins.

Of course, people (even good handicappers) do not agree on the probabilities of horses winning. That is why they bet on horse races. Thus one could in theory plot frequency graphs showing the estimated odds of a given horse winning in a given race, or from those odds, the estimated probabilities.

Of course some handicappers are better than others:

Source

I suspect that you could get handicappers to say that they believed there was a 5050 chance that the probability of a specific horse winning a specific race was between two probabilities p1 and p2. If you  took a number of such judgments you could plot a graph the number of experts who had included winning probabilities. If you weighted the estimates by the known accuracy of the handicapper, you would get something that looked much like a probability distribution over the probabilities of winning.

Now you could begin to make statements like "the odds on this horse" are x with such and such confidence that they are between y and z.

Is this worthwhile for horse races? I don't know. Ask a bookie or a casino.

The approach might be worthwhile for a government making important economic or security decisions.