D.C. court considers how to screen out ‘bad science’ in local trials

There is an interesting article in today's Washington Post titled "D.C. court considers how to screen out ‘bad science’ in local trials".Having managed "peer reviews" of science project proposals for nearly two decades, I have come to the conclusion that this is not an easy thing to do, and even if you have been doing it for some time, you may still get it wrong from time to time;

Fortunately, DC has access to the National Acamey of Science, NIH, NSF. and other organizations with lots of experience doing this, and can delegate the work.

One thing I found useful was developing a data base on how often each expert agreed with other experts. I discovered when I first tried this that expert judgments in that field, which we often taken as 100 percent correct, actually agreed on the final recommendation no more that 90 percent of the time.

In one situation in which the same group of "experts" repeatedly judged similar objects, one of the "expert's" reviews was negatively correlated with all the others. 

On Decisions

Sometimes it is more important that a decision be right, than that you make it.
When you don't have to make a decision now, procrastinate.
If you don't have a decision to make, don't make a decision.

If others have better knowledge for making a decision, defer to their expertise.
If others are better at the analysis for making a decision, defer to their expertise.
If others are wiser, defer to their wisdom.
Know your abilities as compared with those of others.

Some decisions must be made in ignorance;
Some decisions must be made not knowing the probabilities involved;
Some decisions can be made after odds have been calculated;
And of course, some decisions are easy.
If you have to make a decision, see if you can move down the above list.

Some decisions are hard in that we don't even know how they should be framed. Good luck!

We don't always know what we don't know.
We don't always know what is important to the outcome of a decision.
Some of the worst decisions are made neglecting to take into account something that is important, but the decision maker did not realize it was important.

Arrogance is the enemy of good decision making.
People often overestimate their ability to make decisions well.

Really big decisions are likely to be really hard to analyze.

Framing is important; good framing leads to good decision processes and to improve decisions.
Simplistic framing leads to simplistic decisions, that often prove costly.
Excessively complex framing can lead to major difficulties in analysis and thus to poor decisions.
Really hard decision problems may be those in which the information is not available to frame the decision, or to frame it adequately.
Many bad decisions can be traced to badly made frames.

Disinterested, evidence based policy decisions and emotional needs for action

I find myself worrying about the chasm between the way policy should be made and the bases of policy support by the public.

An example in the Bush administration's war decisions. 9/11 was a shock widely shared by the American public. The public after 9/11 would support almost any action to get back at the terrorists who planned the action. The Bush administration, as far as we can tell, made the right decision that Al Qaeda was responsible, and supported those in revolt against the Taliban in Afghanistan on the basis that the Taliban was sheltering Al Qaeda leadership. With the Taliban overthrown, the Al Qaeda leadership was in danger from American retaliation in the east of Afghanistan.

At that point, the Bush administration appears to have misused its intelligence assets and used spurious analysis to argue that Iraqi political leaders had also supported Al Qaeda, were implicated in the 9/11 attack on the World Trade Center, and had stock piles of weapons of mass destruction; all of these charges eventually proved false. However, the public and the Congress, still emotionally driven by anger after the 9/11 attacks, approved of a war on Iraq to be conducted in parallel with the action in Afghanistan, Osama Bin Laden got away in Afghanistan, the U.S. quickly conquered the Iraqi army and eliminated Baath party members from the Iraqi government and military, and then chose to embark on long and costly military efforts in Afghanistan and Iraq.

More American lives were lost in the Iraq and Afghanistan war efforts than were lost in the Twin Towers, thousands more Americans were wounded and/or disabled, and huge numbers spent years of their lives in the war zones away from families and friends. The casualties among Iraqis and Afghans dwarf the America casualties. The cost of the wars have still to be fully counted, but appear to be more than a trillion dollars. Perhaps most important, a large region of the world -- never too stable to begin with -- is now far less stable than it might have been.

My guess is that the Bush administration did not do due diligence in thinking through the causes of 9/11, nor the alternatives available to it, nor the effects of the actions it chose. However, those actions were easy to sell to the American public at the time that they were taken because the public was acting on the basis of emotion and not reason.

Policy should, I believe, be based on a reasoned judgement as to what is best for the nation. Emotional response to current events on the part of the policy makers tends to get in the way of good policy decision making. The public is more prone to emotional response to riveting events, especially those driven home by vivid media coverage, but public intellectuals should strive to keep a dispassionate, evidence based assessment of the situation before the public. Walter Cronkite, the most trusted newsman in America in his time, did yeoman work of that kind anchoring the news around Viet Nam and the Kennedy Assassination.

Good policy making involves drawing on people who have knowledge about the issues and peoples to be affected, and in situations such as planning wars in Iraq and Afghanistan requires reaching out to people outside of the normal framework; there were people in the USA, and indeed in government, who had expert knowledge about these countries, but they were not in the cabinet or the cadre of political appointees in the State Department, Defense Department, nor the Treasury Department. Good policy making would seem to require the president and his key advisers to be able to rise above their emotional responses to immediate events in order to look carefully at evidence and dispassionately consider alternative scenarios and their probabilities. It requires being able to consult with leaders of allies and consider carefully the views of leaders of other countries.

Those who don't know history may repeat it

In the presidential election of 1912, progressive Republican incumbent William Howard Taft sought reelection. Former progressive Republican president Teddy Roosevelt, dissatisfied with Taft's performance, ran for the Bull Moose Party. The split between the Republicans allowed Woodrow Wilson, who also ran as a progressive, to win the presidency for the Democrats.

In his second term. Wilson reluctantly led the country into World War I, but at the end of that war led in the creation of the League of Nations, an international organization conceptualized to help prevent world war. War had become too terrible to countenance.

Republicans controlled the White House from 1923 to 1933, the Senate from 1919 to 1931 and the House of Representatives from 1917 to 1933. Conservatives in the Congress voted against the Treaty of Versailles (which formally ended World War I) and voted against the United States joining the League of Nations.

What Difference Did it Make?

Understanding what would have been had history been different is always dangerous. The State Department writes:

Most historians hold that the League operated much less effectively without U.S. participation than it would have otherwise. However, even while rejecting membership, the Republican Presidents of the period, and their foreign policy architects, agreed with many of its goals. To the extent that Congress allowed, the Harding, Coolidge, and Hoover administrations associated the United States with League efforts on several issues. Constant suspicion in Congress, however, that steady U.S. cooperation with the League would lead to de facto membership prevented a close relationship between Washington and Geneva. Additionally, growing disillusionment with the Treaty of Versailles diminished support for the League in the United States and the international community. Wilson’s insistence that the Covenant be linked to the Treaty was a blunder; over time, the Treaty was discredited as unenforceable, short-sighted, or too extreme in its provisions, and the League’s failure either to enforce or revise it only reinforced U.S. congressional opposition to working with the League under any circumstances.

The isolationism that dominated legislative thinking for much of the time between the wars seems now to have failed to recognize the international role that would be thrust upon the United States and to have contributed to the buildup towards World War II.

It is very important for members of Congress to recognize that they may be wrong in their beliefs and to grant respect to the executive branch views. This is perhaps especially important during the long election season, now at least two years, in which politicians are prone to make statements for political benefit that might not be in the country's best interest.

How Accurate Are DNA Identifications/

As I understand it, courts sometimes use DNA identification to incriminate or clear suspects. Material found at the scene of a crime is found to contain human cells, the DNA is identified, and then compared with the DNA of a suspect.

Popular sources seem to suggest that it is very, very unlikely that two people have identical DNA, or that the wrong person will be identified from a DNA test.

Lets say that the laboratory reports back that the suspect's DNA matches that of the sample from the scene. What does that actually mean?

According to Wikipedia:

Monozygotic twinning occurs in birthing at a rate of about three in every 1000 deliveries worldwide.

So, as I understand it, 0.3 percent of individuals in the population share identical DNA with an identical twin. Unless the existence of such an identical twin can be ruled out, the most one can say is that there in not more than one chance in 333 that someone else left that specific DNA trace at the scene of the crime.

Say that investigators try to rule out the possible existence of a monozygotic sibling. We know that mistakes happen in hospitals and that there are other ways that twins are separated at birth without their ever later learning of that occurrence. So, even if efforts have been made to assure that the individual in question does not have an identical sibling somewhere, that investigation should not be considered absolutely conclusive.

Add in the possibility of error in the lab -- mixing up the crime scene sample with that from another crime scene, mixing up the suspect's sample with that from another person, mistakes in reading the DNA test results, deliberate falsification of the findings. The confidence provided by DNA testimony goes down further.

How about the possibility that the DNA sample taken from the crime scene is not that of the person who committed the crime, but that of some other person? Is it not possible that that does occur? How would you estimate its probability?

How many judges or jurors would be able to really understand the testimony of a DNA witness, properly estimate the probability of a suspect being responsible for the crime from the DNA evidence alone, or able to combine the probabilities arising from the DNA evidence with that from other sources (eye witness testimony, circumstantial evidence, exculpatory evidence offered by the defendant, etc.?

I am not comfortable with the accuracy that the judicial process might achieve in cases involving DNA evidence. Still, such evidence obviously should influence verdicts if available. The question is how to be sure that it appropriately influences verdicts.

A thought about decision making

The best lack all conviction, while the worstAre full of passionate intensity.THE SECOND COMINGWilliam Butler Yeats

Of course, the best do not lack all conviction. They are as convinced as you and I of the current color of the sky or of their own identity. I think the best recognize two classes of situations:

• "Hard Problems" in which either there are no outstandingly suitable solutions, or a great deal of work remains to be done to find such a solution;
• Crises, in which the full set of options have not yet become visible and/or adequate information is not yet available to select the best option.

In such circumstances, the best decision makers would seldom if ever "bet the bank" on a guess. Lacking conviction is often the best means of coming to a good resolution of a situation that one would categorize as a hard problems or a crisis. Indeed, in such situations the best practice is sometimes:

• To estimate if some action is required immediately:
• If so, take a modest action, ideally one that can be retracted if it proves important to do;
• If not, spend some time and effort gaining more information, identifying more options, considering more incremental steps, and evaluating those steps.

Unfortunately, this (which seems to me the best approach) is often termed "muddling through", a term which to many suggests weakness.

Talking heads on television often seems full of passionate intensity, secure in the knowledge that they have the unique understanding of the issues and recognize the best of all responses to the question at hand. These folk may make for good television, but it seems to me that:

• TV broadcasters find it relatively easy to find two or more people full of passionate intensity who differ radically on their understanding 
• If you watch for a few years, you begin to understand that the talking heads are often wrong, that situations are often more complex than they realized, that they lacked key items of information, or that they had failed to think about options that eventually proved important and useful.

I note, following Michael Bohn, that presidents in crisis seem to lack all conviction when they make good decisions in crisis situations, but presidential candidates seem full of passionate intensity when they advocate positions or criticize their opponents. Good decision making seems to benefit from listening to all points of view, being open to new information and new options, and deferring decisions until it is worse to do than to act prudently.

Crowd sourcing as an aid to medical diagnosis

There is an interesting article in The Economist describing CrowdMed, a website that uses crowd sourcing to help diagnose rare diseases.

The need for a “crowdsourced” service like this comes from the number of rare diseases around. The National Institutes of Health, America’s medical agency, recognises 7,000—defined as those that each affect fewer than 200,000 people. A general practitioner cannot possibly recognise all of these. Moreover, it may not be clear to him, even when he knows he cannot help, what sort of specialist the patient should be referred to. Research published in 2013, in the Journal of Rare Disorders, says about 8% of Americans—some 25m people—are affected by rare diseases, and that it takes an average of 7½ years to get a diagnosis. Even in Britain, with all the resources of the country’s National Health Service at a GP’s disposal, rare-disease diagnosis takes an average of 5½ years. Also, doctors often get it wrong. A survey of eight rare diseases in Europe found that around 40% of patients received an erroneous diagnosis at first. This is something that can lead to life-threatening complications.

My wife last year went through the process of getting a diagnosis for a rare set of symptoms, and it took a while, involved many tests, and ultimately a referral to a specialist. I am impressed by the problem of diagnosis for such conditions, and I think crowd sourcing could be useful in the right hands.

I would caution that it could be dangerous as well. I think that one should use CrowdMed only under the care of a physician, and should discuss possible diagnoses with the physician.

Still, I find the idea of a website providing an alternative source of ideas to be discussed with one's doctor to be very interesting.

A Thought on Decision Making in White House Crisis Situations

Michael Bohn recently did a TV book discussion on his new book, Presidents in Crisis: Tough Decisions inside the White House from Truman to Obama. Bohm was the naval officer (i.e. non-political) man in charge of the White House situation room during the Reagan administration. He writes about crisis, in the sense that there arise suddenly, and the president has to decide if the U.S. is to respond, and if so, how. Bohm notes that the decision as to whether to hold meetings to discuss the president's decision in the Oval Office (or presumably the Cabinet Room) or the Situation Room is usually a political one, meant to give a message to the press and the public. Bohn, because of his position running the situation room, got great access to people who participated in 17 discussions treated in the book (including some presidents).

He made a couple of general points that we should all recognize. First, unless you have been there, you have no idea how hard it is to make decisions like how to respond to an invasion by or on an ally. People not actually involved don't know all the ramifications and don't have all of the date, and don't have the responsibility. The party in opposition to the president will have spokespersons trying to make political points when such a situation occurs, but Bohn would not pay attention to their comments. He noted that Reagan, while running for president said he would be bold and active in such decision making, while as president he was at his best when slow and cautious.

Secondly, Bohn said that domestic politics always counted in the 17 decisions he studied (and they counted more in election years). In the last year of President Johnson' term, since he was not running for another term, they counted less.

Bohn notes that there seems always to be a significant lack of information in such crisis decision making ("fog of war"). He strongly recommended incremental decision making -- one small step at a time, allowing the president to step back if necessary. In the time between incremental decisions, new information may become available and new options may develop. He also recommended that serious efforts be made to avoid "group think" by assuring that people with different positions are at the table; I assume that early on, when there is little information and few options, the president might seek people with different general viewpoints (e.g. hawks vs. doves, local expertise vs. general foreign policy overview).

Of course, in foreign policy crises, there are not only domestic viewpoints to be taken into account, but also foreign ones. Consider for example the current situation involving ISIL, Syria and Iraq. There are of course different parties in Syria and Iraq, with different interests, and I suppose different parties within ISIL. Then there are the many neighboring states that feel that they may have important interests in the conflict. Then there are our European and Asian allies and Russia. If one considers the United Nations, the 193 member states may all vote on General Assembly resolutions, and each of them may have different parties with distinct views, even if the member state seems far away and little affected by a war in the Levant.

It occurs to me that there might be a some relevance to chess or other games of strategy. In chess, the game is too complicated for even a computer to calculate all alternative sets of possible moves to chose the best one at each point. Muddling through (see Charles Lindblom's classic "The Science of Muddling Through") involves the strategy advocated by Bohn -- making one move at a time and waiting to see what happens and what others do before committing to the next move. It is the classic approach in chess. Still, a good chess player, when he/she opens the game first, chooses a gambit; his/her opponent will normally recognize the gambit and select an appropriate response gambit. Thus good chess players depend on their experience with the game to have opening moves that hold little surprise for their opponents, but that lead toward known positions that are strong for the middle game. Decision making in crisis situations might well lend themselves to such series of initial moves at least between countries that frequently face each other in such situations.

An important part of crisis decision making is simplifying the situation in such a way that its important aspects are maintained, superfluous issues are ignored, and it becomes possible to move ahead reasonably expeditiously. Doing this well seems very hard to me, yet  it seems to be a critical skill. Thinking of chess again, expert players have developed intuition, based on experience, that allows them to focus on a few alternatives at each step to analyze in depth; they know when to rely on memorized alternatives, when shallow analysis is adequate and they can save time and effort and when to take the time and put in the effort to analyze in depth before making the next move. That intuition is I think based on replying many games by experts, reading analyses by experts of influential games and matches, and playing a lot.

Good decision makers also delegate a lot of the decisions. Thus the political leaders of the Allies in World War II made the policy decisions, but delegated the responsibilities of figuring out how to carry out those policies to military and civilian mobilization leaders. These latter leaders in turn made the strategic decisions on how to implement the policies that they were given, but delegated tactical decisions to their subordinates.

I think, in a well run White House, decision makers fully utilize the structures of government to get information and have it screened and manicured to assist in their incremental decision making. Thus, the State Department has thousands of expert foreign service officers stationed around the world and a world class system to filter their reports and provide a regular synthesis to the Secretary of State and senior staff, who in turn provide that information to the National Security Council and the President. Similar systems exist in the Defense Department, the Department of Commerce, the Treasury Department and the rest of government, according to the charter of each agency.

Government decision making works well not only when the decision makers themselves are capable, but also requires effective agencies to carry out delegated tasks and effective agencies to evaluate results of decisions and obtain new information.

“This is how fast America changes its mind”

Chris Blattman shared this graph that he picked up from Bloomberg

Juries Should Accept Testimony "With a Grain of Salt"

The Washington Post yesterday published an article (from which the above graphic was drawn). I quote from its lead paragraph:

The Justice Department and FBI have formally acknowledged that nearly every examiner in an elite FBI forensic unit gave flawed testimony in almost all trials in which they offered evidence against criminal defendants over more than a two-decade period before 2000.

 The FBI has spent decades seeking to establish a reputation for expert analysis of evidence and fairness in the presentation of its results. The halo effect on juries is not accidental.

Is fingerprint identification accurate and reliable? Here is a quote from one source:

The handful of studies of fingerprints show a troubling pattern of errors. Since 1995, Collaborative Testing Services, a company that evaluates the reliability and performance of fingerprint labs, has administered an annual and voluntary test. It sends fingerprint labs a test that includes eight to twelve pairs of prints that examiners confirm or reject as matches. The pairs usually consist of complete, not partial prints, making identifications easier than the real situations examiners face. Nevertheless the error rate has varied from 3% to a dismal 20%. 

Equally troubling is a test conducted by the FBI. During Byron Mitchell’s trial for armed robbery in 1999, his lawyers questioned the reliability of fingerprint identifications. In response, the FBI sent two prints taken from the getaway car and Mitchell’s prints to 53 crime labs to confirm the agency’s identification. Of the 39 labs that sent back their results, 9 (23%) concluded that Mitchell’s prints did not match those from the car. The judge nevertheless rejected the defense’s challenge and accepted the fingerprint evidence. Mitchell was convicted and remains in prison. The FBI has not repeated the experiment.

Lie Detector tests were once used widely by police and as evidence in courts. Here is what one source says about them:

In past times they were often used by police and government agents to interrogate suspected criminals, but as they have been proven to be extremely unreliable indicators of lying, their use has lessened in recent years (usually because of court rules prohibiting their use).

How about eye witness testimony? I draw this from an American Psychological Association website:

"Like trace evidence, eyewitness evidence can be contaminated, lost, destroyed or otherwise made to produce results that can lead to an incorrect reconstruction of the crime," he says. Investigators who employ a scientific model to collect, analyze and interpret eyewitness evidence may avoid incidents like Olson's potentially flawed identification of the Fairbanks suspects, he notes. 

In fact, Wells says that other evidence techniques, such as police lineups, are similar to scientific experiments. In lineups, the police have a hypothesis, they provide instructions, collect responses and interpret the results. As such, the same factors that can bias the results of an experiment can bias an eyewitness's performance in picking suspects out of a lineup, he says. 

In a 2000 American Psychologist (Vol. 55, No. 6, pages 581-598) article that dovetailed with the Department of Justice report, Wells and his colleagues outlined a number of ways police can avoid biasing eyewitness testimony, including warning the witness that the actual perpetrator may or may not be in a lineup, maintaining a double-blind lineup environment so that a detective cannot influence a witness's judgments and securing a statement of the witness's certainty of their identification. 

"To a research psychologist, the changes seem obvious," he says. "But to law enforcement officials they are very different from how they've tended to do things in the past."

I wonder how many jurors understand that testimony that they hear in court, even expert testimony, may not be accurate. How many are prepared to recognize that even FBI testimony may be flawed and may not in itself "prove" culpability. 

Are American Juriies Adequately Educated to Do What They Are Asked to Do?

American juries are asked to decide in criminal cases whether the state has proven its case beyond a reasonable doubt and in civil cases which side has the preponderance of evidence in its favor. I wonder how many Americans are really capable of making such sophisticated judgments. Do jurors more generally decide which side they "believe", as the two sides present their different narratives of the case at hand?

I spent years thinking in terms of Bayesian statistics -- how each piece of evidence changed the probabilities of one or another hypothesis. I am quite comfortable not "believing" either of two narratives explaining the same outcome. Indeed, I think it quite reasonable to assume that one might -- in principle if not in fact -- assign a probability that narrative A is true, a probability that narrative B is true, and a probability that neither narrative A nor narrative B is true.

I am also familiar in a general way with the literature on common cognitive biases I would guess I am more familiar the existence of such biases than the vast majority of Americans.

I have had some experience, documented in my doctoral dissertation, with the errors that experts make in estimating their own accuracy, and in statistical means for estimating accuracy of expert judgment.

I have also been involved in the review of many thousands of research proposals and fellowship applications, and have many times participated in decisions as to whether a review was or was not adequate, based on the background of the reviewers and observation of panel behavior.

I assume that the "preponderance of evidence" means that a jurist is to decide in favor of the narrative he/she thinks is more probable based on the evidence provided.

I an not sure how great a probability for the narrative provided by the state is required for a jurist to find that the defendant is guilty "beyond a reasonable doubt".

My basic point is that it does not seem necessary nor appropriate to "believe" one or the other narrative. It is possible and indeed preferable to consider the probability one would assign to each narrative to be true.

On getting good information from peer review meetings

I read an interesting column in The Economist on getting good results from meetings. I quote extensively from that Free Exchange column:

In 1785 the Marquis de Condorcet, a French mathematician and philosopher, noted that if every voter in a group has a better-than-even chance of choosing the preferable of two options, and if voters do not influence each other, then large groups of voters are very likely to make the right choice.* The bigger and more diverse the group the better: more people bring more information to the table which, if properly harnessed, leads to improved decisions. But ever bigger meetings imply more time spent in them: few workers would welcome that. And even with more people in the room, all manner of behavioural flaws stand in the way. 

One problem that obstructs sensible decision-making is the “halo effect”—“owning the room” in the parlance of Silicon Valley.......A second problem is called “anchoring”. In a classic study Amos Tversky and Daniel Kahneman secretly fixed a roulette wheel to land on either 10 or 65. The researchers span the wheel before their subjects, who were then asked to guess the percentage of members of the United Nations that were in Africa. Participants were influenced by irrelevant information: the average guess after a spin of 10 was 25%; for a spin of 65, it was 45%. In meetings, anchoring leads to a first-mover advantage. Discussions will focus on the first suggestions (especially if early speakers benefit from a halo effect, too). Mr Kahneman recommends that to overcome this, every participant should write a brief summary of their position and circulate it prior to the discussion...... 

Tom Gole of the Boston Consulting Group and Simon Quinn of Oxford University studied the votes of judges at international school debating tournaments. In these tournaments, three judges are randomly assigned to each debate (with some controls to ensure each panel has a mixture of experience, gender and so on). Judges watch the debate, then immediately vote on the winner before conferring. Crucially for the experiment, the participating teams are seeded. Using some whizzy statistical modelling, the authors find that if a judge disagrees with her fellow panellists in a given round, she is more likely to vote for the pre-tournament favourite—the higher-seeded team—in later debates. That suggests, say the authors, an unspoken desire to avoid “too much” disagreement. 

Career concerns may distort incentives even if votes are secret. In a 2007 paper Gilat Levy of the London School of Economics noted that observers can work out how likely it is that committee members have voted one way or another from ballot rules. If unanimity is required for a measure to pass, and it does, then outsiders will know with certainty how every member has voted. A simple majority rule means that observers can assign at least a 50% probability to any one committee member having given their assent; if a majority of two-thirds is required, the probability that any given member has supported the proposal goes up. The incentive to vote against controversial measures rises the greater the likelihood that each member will be blamed for its passage.

I have posted a number of times on peer review (check the "peer review" tag to get them all. What we did to get good results in the peer review of research proposals was:

• To try to screen at a preproposal level to be sure we were asking reviewers to look at decent proposals (and to save proponents from writing detailed proposals with little chance of being funded).
• Criteria for approval of proposals were carefully defined and described. (Note that the purpose of the program determines the criteria; our program was intended to build capacity to do research in developing countries, and sometimes a proposal strong on this criterion would be funded while a stronger scientific proposal with less capacity building involved would not be funded.
• To group proposals into small groups on related topics, and send each to a primary and secondary reviewer. They would be asked to write a review and rate the proposal prior to the meeting.
• The primary and secondary reviewers for all the proposals in a single group would form a panel. All the proposals would be sent to each member, and all the members of the panel were expected to read all of the proposals before the face-to-face panel meeting. (Normally, something like 8 to 10 proposals would be reviewed by a single panel, and normally a single reviewer would not be the primary reviewer on more than two proposals, nor the secondary reviewer on more than two.)
• In the meeting, the proposals would be discussed one at a time. The primary reviewer would speak first on a given proposal, and then the secondary reviewer would speak. There would then be a general discussion, after which, the proposal would be rated by all of the reviewers individually. The average rating would be calculated. Proposals at this stage would also be judged by consensus to be potentially worthy of funding or so flawed in some way as not to be worth funding.
• When all the proposals had been reviewed, discussed and rated, they would be ranked from best to least useful by consensus of the panel. The average rating would be a guide to the ranking, but we discovered that sometimes after discussion a panel would find that comparing two proposals they would rank that with the lower rating to be in fact more worthy of funding and thus of higher rank.
• At all panel meetings we would have at least two people to judge the panel discussion -- one from our office and one from the National Academy of Sciences staff. If the judges determined that the review was not adequate for some reason for one or more of the proposals, additional reviews would be sought and a final decision would be taken by our staff.
• We would then allocate our total budget among the top rated projects (all of course specified as worthy of funding by their respective panels), using the ratings as a guide as well as the observations of the panel reviews.

The program was part of the foreign assistance effort of the U.S. Government, and the reviews were held in the Washington D.C. area, which has a very large and strong scientific community. We found that local scientists were willing to do the reviews and come to peer review meetings without fees or reimbursement of costs if we selected from the local universities, laboratories or government agencies. Ours was a relatively small program. Reviewers seemed to enjoy the experience.

Decision Making Leading Up to the Civil War

Source

The Civil War took place from 1861 to 1865. As the commemoration is coming to a close, let me share an imaginary conversation I have in my mind taking place in 1866 or 67 with someone who was in the South Carolina legislature when it took the decision to secede from the Union in 1861.

     Me: I understand you were there in the state legislature when the vote was taken to secede from the Union.

     Respondent: Yes.

     Me: And you were there when it was decided to fire on Ft. Sumter and start the Civil War.

     Respondent: Yes

     Me. How did those decisions work out for you and your colleagues?

     Respondent: (Silence, tears in his eyes.)

Of course a generation of young Southern men had been decimated, slavery had been abolished suddenly and without compensation, Sherman's troops had marched through South Carolina leaving devastation in their wake, there had been few exports for years, and the economy and much of the infrastructure of the state was in ruins. Federal troops were garrisoned in the state, and the ruling White plutocracy had been dethroned.

Virginia had seceded from the Union after much debate and a significant pause. Much of the fighting during the war had taken place in Virginia, and West Virginia had been allowed to secede from Virginia; if anything the impact of the war was worse in Virginia than in South Carolina. Its former plutocrats had perhaps more to lament than those in other formerly Confederated states.

It seems to me that people in power in the South made a big mistake in choosing to secede from the Union and fight a war with the northern states of the Union. The independent decisions of South Carolina and Virginia politicians were both mistakes. Indeed, it seems to me that the decision making process must have been faulty; they must have failed to accurately assess the probabilities of alternative courses of the war, the results of each course, and thus the probable cost of secession and war.

Even had the Confederacy succeeded, that success probably would have been partial and at great cost to the South. What if the Union had made peace with the Confederacy, but held control of significant amounts of territory that had been part of the pre-Civil War southern states, and held control of the Mississippi? How would a small nation, with an agricultural economy lacking in industry and sea power, recovering from war, have done in the imperial age? Slavery would have been abolished eventually as happened in all of the Americas, and as the European imperial powers colonized Africa, the markets for cotton and other agricultural exports would have become more competitive. The French showed in putting Maximilian and Carlota on the throne of Mexico, that they would snap up American influence given the chance.

In contrast, it seems to me that the people in the north made the right decision. The north had sufficient advantage in manpower, economic power and diplomatic power to win the war. The European powers might want the Confederacy to succeed, but would find it hard to intervene visibly to support those fighting a war to continue slavery. It was also important to save the Union, since only a strong union would rise to quickly become a world power (with great benefits to future generations). Moreover, it had the moral arguments on its side -- it was important to show that democracy could work and develop, and it was right to end slavery.

Of course, there were many possible courses for the war. Seven slave holding states seceded in rapid succession from December 20, 1860 to February 1, 1861. Four more states seceded from the Union between April 17, 1861 and June 8, 1861. Four border states and the District of Colombia, in which slavery was still legal in 1861, remained in the Union. More or fewer states might have seceded; fewer might have meant a shorter war, more a longer war and a greater possibility of success of the Confederacy. Although a priori unlikely, Great Britain might have sided with the Confederacy, and France might have chosen to do so rather than chance a power play in Mexico.

What would have been the course of history had the South Carolina and Virginia not seceded, but rather had bargained for terms to stay in the Union? It seems likely that such a bargain could have averted war, but would have required the abolition of slavery; I would guess that the abolition could have been more gradual, with some manner of reducing the economic impact on the plantation economy of the south.

Huge advances in decision science were made in the 20th century, and it is not fair to expect politicians in the 1860s to do decision analysis as well as it can be done today. On the other hand, watching the political process in the USA and in other countries, it seems clear that irrational decisions are still common -- and indeed they rule the day.

A Thought About Freedom of Speech and the Great Dissent

I am reading The Great Dissent: How Oliver Wendell Holmes Changed His Mind--and Changed the History of Free Speech in America by Thomas Healy. The book seeks to tell the story of how the Supreme Court Justice changed his mind about the limits that the government could impose on free speech, moving from a position permissive of the government's laws against sedition to one much more in defense of the citizen's right to speak his/her mind.

I think there was always an understanding that one could not lie about someone thereby damaging him or her, and that the government could impose penalties (observing due process) on one who broke the laws against libel or slander. Holmes appears to have loved discussion of ideas and to have recognized that ideas would be clarified and improved in the give and take of open discussion. In the context of World War I, the Congress limited freedom of speech in legislation primarily focused on stopping espionage. Holmes began with the belief that it was permissible for the government to limit speech opposed to the war effort, but came to feel that the Supreme Court should be explicit in indicating the standard to be used in deciding which speech could be limited and which must be free.

In reading the book, it occurs to me that in 1919 few would have had a good idea on how to qualify risk. Thus, one might have tried to estimate the risk of someone's speech leading to injuries or death. In today's language, one might look at the expected number of disability adjusted life years (DALYs) lost due to an utterance. That value could be quite large if for example someone revealed to the enemy the plan of attack of a new offensive on the western front. I read today that doctors in Kenya are spreading the rumor that immunization is being done with an additive to the vaccine that sterilizes the recipient; that is a speech that is likely to have a significant impact measured in DALYs. On the other hand, a person with little influence railing against a well accepted government policy might have cause negligible risk by that speech.

Of course, balancing the cost one might also consider the potential benefits of the act of speech. Holmes famously wrote that there was not right to yell fire in a crowded theater. Of course, there is no such right if there is no fire, but every right if there is a fire and alerting the audience will save lives. What if someone makes some observations which lead him/her to think that there is a probability of a serious fire in the theater, but does not lead to certainty; assume further that an immediate warning would be effective in avoiding injury and death, but waiting to be sure would lose the opportunity to do good. One could in theory again calculate the likely benefit of yelling "fire". Would that justify the yell if it proved later that there had not been a fire?

A Higher Level Risk-Benefit Analysis

What would be the cost in DALYs of each of the various standards that the Supreme Court might set for the nation as the standard became accepted and used nationwide? What would be the cost in enforcement in accordance with that standard. These would seem to be relevant for the justices to consider in setting such a standard.

On the other hand, what would be the larger cost to the nation in the limitation on free speech -- which is after all a basic freedom on which the nation was founded. How much would the public debate be affected, and the process of public policy determination hurt?

It is less than a week after the election, and this election seems to have occasioned a flood of negative advertising. Was any of this false advertising? A lot? A little? Should there be some limitation on politicians making false or misleading statements in an election campaign? Should there be a limitation on others making such statements verbally or in the media during a campaign? I don't know, but I wonder whether the judicial process has access to adequate analytic expertise to properly analyse the issue.

The Roslings suggest we should make fact based decisions

Four heuristics that help in responding to questions about the world:

1. Things are getting better, not worse.
2. The world is no longer so divided between haves and have-nots; rather there is a single node distribution of income, with the poor at one tail and the rich at the other.
3. Social advances are coming before income increases; the majority of people in the world are already enjoying many of the benefits we tend to associate with development.
4. You overestimate the probability of harm from things you are most afraid of (and underestimate the danger of things you are accustomed to living with); you overestimate how good you are at dealing with every day dangers.

People make bad decisions every day because they don't understand reality. I like the suggestion that we don't understand global reality because of:

1. Personal Bias: we grew up and live now in places that are unrepresentative.
2. Outdated Facts: the things we learned in school are no longer true of today's world, nor are the things in books printed earlier than yesterday (i.e. all the books we actually read).
3. News Bias: news media provide "news" that interests people rather than trying to describe the actual state of the world.

Of course no reader of this blog would fall into any of these traps (as I do all the time). But climate deniers, those who fear genetic engineering is a greater threat to the world than hunger, those who don't immunize their children because they have heard of (very rare) complications or imagined complications and others fall victim to these biases.

You are in a lot more danger in the USA from a gun owned by a good American and from a driver who considers himself safe than you are from a foreign terrorist.

Research on forecasting.

Tetlock also found that specialists are not significantly more reliable than non-specialists in guessing what is going to happen in the region they study. Knowing a little might make someone a more reliable forecaster, but Tetlock found that knowing a lot can actually make a person less reliable. “We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly,” he reports. “In this age of academic hyperspecialization, there is no reason for supposing that contributors to top journals—distinguished political scientists, area study specialists, economists, and so on—are any better than journalists or attentive readers of the New York Times in ‘reading’ emerging situations.” And the more famous the forecaster the more overblown the forecasts. “Experts in demand,” Tetlock says, “were more overconfident than their colleagues who eked out existences far from the limelight.”

The quotation is from a review of Philip Tetlock's book,  Expert Political Judgment: How Good Is It? How Can We Know?, from the New Yorker magazine.

Some years ago I compared the forecasts of GDP made by World Bank economists with simple linear extrapolations of GDP. On average, the linear extrapolations were slightly more accurate. Now economists realize that there is a great deal of inertia in national economic systems, so they tend to be relatively conservative in forecasting deviations from the trend. My data seemed to show that when they did forecast a significant change from the trend, they tended to overestimate the actual deviation. I have known and worked with several World Bank economists, and I have the greatest respect for their competence. I think my findings were consistent with Tetlock's view that experts are frequently not as good at forecasts as we might expect from their frequent appearances in televised news programs.

Another quote from the New Yorker:

Low scorers look like hedgehogs: thinkers who “know one big thing,” aggressively extend the explanatory reach of that one big thing into new domains, display bristly impatience with those who “do not get it,” and express considerable confidence that they are already pretty proficient forecasters, at least in the long term. High scorers look like foxes: thinkers who know many small things (tricks of their trade), are skeptical of grand schemes, see explanation and prediction not as deductive exercises but rather as exercises in flexible “ad hocery” that require stitching together diverse sources of information, and are rather diffident about their own forecasting prowess.

Foreign affairs are complicated. There seem to be high probabilities of making mistakes in predictions on what is going to happen in Ukraine, Syria, Iraq and other hot spots. Perhaps it is best to make small decisions and not work from a grand plan (one big thing) in foreign policy. Certainly it seems wise to look for the "black swan", the event or piece of information that doesn't fit with forecasts and assumptions.

Tetlock also has an unscientific point to make, which is that “we as a society would be better off if participants in policy debates stated their beliefs in testable forms”—that is, as probabilities—“monitored their forecasting performance, and honored their reputational bets.” He thinks that we’re suffering from our primitive attraction to deterministic, overconfident hedgehogs.

I wonder whether Tetlock is a hedgehog or a fox? How good is his intuition as to what would improve our society? 

A thought about causality in history

I have been thinking about World War I, and just read The Sleepwalkers: How Europe Went to War in 1914 by Christopher Clark and the wonderful Introduction in The Long Shadow: The Legacies of the Great War in the Twentieth Century by David Reynolds. I have also been interested in the Civil War which the country has been commemorating since 2011. It occurs to me that the two offer a means to say something about causality in history that might be interesting.

Bombardment of Fort Sumter

The Civil War

The "original sin" of the United States was slavery. There was a world wide historical process of abolition of abolition of slavery. It had a variety of causes, including a broadening consensus that slavery was morally wrong, and an increasing consensus that free labor was more productive than slave labor and a better basis for building an economy. The eradication of America's original sin seems to have been part of this historical process.

Some countries abolished slavery in civil war or insurgency (e.g. the United States, Haiti, France) while some did so peacefully (e.g. the British Empire, the former Spanish colonies in the Americas, Brazil). Thus I conclude that the U.S. Civil War was partially caused by the desire to abolish slavery and partially by the failure to find a peaceful way to do so.

The proximate cause of the Civil War was the decision of state governments of southern states to secede from the Union and fire on a Union fort in Charleston harbor, followed by the decision of the Federal government to fight to preserve the Union. My reading of history indicates that the Union was intended from its first government to be perpetual, and that the Union was more populous, richer and more militarily powerful than the Confederacy and very likely to win the war.

I also conclude that the southern leaders made a disastrously bad decision to go to war. Four years later, slavery was abolished, the southern economy was in ruins, and hundreds of thousands of its young men had been killed or wounded. Thus the proximate cause of the Civil War seems to me to have been a disastrous mistake made by the leaders of South Carolina and other states that seceded from the Union. They surely could have found a peaceful path (as was done in other countries) that would have been better for their states, their people and themselves.

Let me suggest then that an intermediate cause was the way political power had been institutionalized in the south. It was the owners of large plantations with large numbers of slaves who were the richest people in the south. They had the time, resources and motivation to control the political institutions, and did so. However, they perhaps tended to put the interests of their class before that of the rest of the people of their states; moreover, the small population from which the leadership was drawn did not produce the quality of leaders needed to deal with the crisis presented by the results of the election of 1860.

Assassination of Arch Duke Franz Ferdinand and his Wife

World War I

World War I was a war between coalitions of empires -- Austria-Hungary, Germany and the Ottoman Empire versus Russia, the French, British and Italian Empires, and eventually the United States. Most of these were multi-ethnic empires, held together by a central authority that had monopoly control of naval and military power, transportation and communications within the empire.

The 20th century saw decolonization and destruction of these empires (even the United States gave up the Philippines). Thus there was a historical process which was tending to destroy the imperial institutions of empire. The institutional form appears to have been historically unstable.

Could the empires have been dissolved peacefully? While tzarist Russia was overthrown by revolution, the Soviet Union broke up relatively peacefully. While the German, Austro-Hungarian and Ottoman Empires were dismembered when the lost World War I, creation of the British Commonwealth after World War I and decolonization of British colonies in India and Africa after World War II were relatively peaceful. So yes, in principle, empires could have been broken up without the mass violence of two world wars.

The proximate cause of World War I was the declaration of war by Austria-Hungary on Serbia, the subsequent declaration of war by Russia on Austria-Hungary and Germany, which in turn triggered more declarations as countries honored treaty obligations. All of this was a response to the assassination of the Grand Duke and Duchess of Austria-Hungary.

Clearly the war was a disaster for the Habsburg government of Austria-Hungary, the Romanov government of Russia and the Hohenzollern government of Germany, and certainly the assassination did not require the world war, but could have been resolved by other means that would at least have allowed the prime mover governments to survive longer and have time to find a better path to accede to the historical trends of history. Thus, I suggest that the proximate cause of the war was the poor decision by these three governments to go to war rather than to find an alternative course that would have preserved them in power longer and allowed more time to find a solution to the problems cased by the decline of the Ottoman Empire and the radical restructuring of the governance in the Balkans.

An intermediate cause of the war may have been the way governance was institutionalized in these empires. Monarchy could and did put people at the head of state and government who were not able to manage the affairs of large empires well; narrow aristocratic classes from which policy leadership was drawn also proved unequal to the job of running empires and failed in the crisis of 1914.

Final comment

Perhaps it is useful to look for long term historical processes that underlie major events, as the abolition of slavery drove the Civil War and as the abolition of 19th century style multi-ethnic empires dominated by a metropolitan central power and monarchy drove the World Wars of the 20th century.

When a decision leads to major disaster for the people making the decisions (as the decision of South Carolina leaders to start the Civil War and the decision of the Austria-Hungary, Russian and German imperial governments to go to start World War II did), perhaps one should look at failures in decision making as proximate causes. Even these are complex, involving the institutionalization of the governmental decision making, the failure to predict the actions of others, and erroneous perception of the relative capabilities of one's own side and the difficulty of the task being considered.

In the two examples, I have suggested an intermediate cause -- an institution of power structures dominated by narrow elites that fail to recognize and foster the interests of the populations that they rule, and that failed to put leaders in place that could make good decisions in the face of existential crises.

I suppose that in both cases the evolution of the technological infrastructure led to a geographic expansion of the state, a growth in the population of the state, a change in the economic structure of the state and an increase in the pace of change. New economic and political institutions had to be developed to better manage the changed socio-economic environments.

Reminding Ourselves that our schools could do better

A graphic released with the 2012 PISA results shows the annualized change in performance in average math scores between 2003 and 2012. The chart includes only nations that have comparable data from both 2003 and 2012.

I quote from the NPR article:

American 15-year-olds continue to turn in flat results in a test that measures students' proficiency in reading, math and science worldwide, failing to crack the global top 20. 

The Program for International Student Assessment, or PISA, collects test results from 65 countries for its rankings, which come out every three years. The latest results, from 2012, show that U.S. students ranked below average in math among the world's most-developed countries. They were close to average in science and reading. 

"In mathematics, 29 nations and other jurisdictions outperformed the United States by a statistically significant margin, up from 23 three years ago," reports Education Week. "In science, 22 education systems scored above the U.S. average, up from 18 in 2009."

The United States is a big country, and our schools are better in some regions than in others. The average is pulled down by the kids in those areas with schools that are not so good. But then so too is the future productivity of our workforce and success of our economy. A mind is a terrible thing to waste! 

Thinking about causes and those responsible for events

Can we generalize from the ways police and courts deal with evidence?

I enjoy watching the police procedural set in the Caribbean, Death in Paradise. In each episode of Series One, Inspector Poole lists all of the possible suspects for the murder under investigation, and then checks off whether each has motive, means and opportunity. This is the standard recognition that if a person is responsible for an event (in this case, a crime), then that person must have the means and opportunity for causing the event (crime) and probably has a motive for doing so.

This is related to the reporters focus on answering the questions, who, what, why, where, when and how? The policeman knows what, where and when to begin with, and focuses on the who, why and how.

So how do detectives prove a case?

• They get evidence, such as videos showing the commission of the crime, fingerprints proving that a suspect was at the scene of the crime, etc. Of course, such evidence is usually not conclusive. Videos may not suffice to identify a person uniquely, and fingerprints may be left at a time other than when the crime is committed.
• Eye witnesses, but we know that eye witness testimony is quite fallible.
• Incrimination of the suspect from others who are known to have been involved in the crime, but sometimes such testimony is false and self-serving.
• Confession, but even confessions to crimes may not be truthful indications of who committed the crime.

It is thus in the courts that juries or judges are asked to decide whether the evidence. Here is a statement of the standard:

The prosecutor must convince the judge or jury of a defendant's guilt "beyond a reasonable doubt." This burden of proof is the highest that our system of justice imposes on a party to a trial. By contrast, in civil cases, such as those seeking damages for personal injuries or breach of contract, a plaintiff's burden is to prove a defendant liable by a  preponderance of the evidence -- just over 50%. 

As a practical matter, the high burden of proof in criminal cases means that judges and jurors are supposed to resolve all doubts about a defendant's guilt in favor of the defendant. With such a high standard imposed on the prosecutor, a defendant's most common defense is to argue that there is reasonable doubt -- that is, to argue that the prosecutor hasn't proved beyond a reasonable doubt that the defendant is guilty. 

I suggest that these ideas provide a checklist that might be used in trying to understand situations in your own life, perhaps especially your business life when things happen that affect you that you are having trouble understanding. 

Why did Civil War soldiers fight?

I watched Christopher Hamner deliver a lecture on the motivation of Civil War soldiers on television. It was part of the regular series of lectures on American History TV. I was especially taken by the discussion with his students of why soldiers fought in the war. The students did a good job, with his coaching and based on readings assigned for the class, in giving many reasons -- patriotism and ideology, religious conviction (doing God's will), community pressure, seeking adventure, money, etc,

Professor Hamner suggested that there were different decisions made at different times, that different soldiers may have had different reasons, that the reasons for any specific soldier's decision would be complex (and indeed I think it was implicit in the discussion that any individual soldier might fail to fully understand his own decision).

Still as I reviewed the lecture in my mind, I came to the conclusion that many of the key reasons might have been missed.

I suspect that it is instinctive for young men to step forward to fight for their community. Homo sapiens are a social species and I suspect that the species evolved with young adult males defending their social bands -- they would have been the strongest and the most experienced using weapons. Thus I suspect that young men stepping forth to do battle to defend their families and communities is instinctive -- something that comes from unconscious drives. (We are also a species that evolved to be successful by using our brains, and probably the older males in ancient bands organized their defense when necessary.)  I further suspect that all societies have institutionalized mechanisms to harness this instinctive behavior to the preservation of their own social order.

Let me address some additions to the student proposals specific to each of Hamner's three big questions about soldiers in the Civil War by adding some reasons that were not mentioned in his classroom.

Why did they enlist in the military?

• Young men frequently do foolish things, perhaps due to an excess of animal spirits, alcohol or testosterone.
• They may have underestimated how hard it would be to serve, how great the risks were in service, or how great the rewards would be from that service; they may have done so because they were misled by their elders.
• The may have failed to properly consider alternatives to military service, such as emigration to Canada, Mexico or California.

Why did they stay in the military instead of deserting?

• The task of escaping from ones unit, traveling away from the front lines to a place of safety, and reestablishing a civilian life may have been too daunting.
• Many did in fact desert and successfully return to civilian life.

Why did they go into battle, especially into the horrific battles later in the Civil War when at least the experienced soldiers knew what they were in for?

• Some soldiers like battle.
• Some soldiers may have so hated the enemy after seeing their comrades wounded, maimed and killed and after fighting those enemies for years that they were willing to endure battle for the opportunity to attack the enemy and obtain revenge.
• Fatalism.

Three more responses to this last question involve more complex answers.

• They were programmed to do so. I recall reading that a British officer in the War of 1812 said that it took three years of training to produce a good body of professional soldiers. He would have been thinking of soldiers who could fight in formation, firing rapid volleys, and standing up to the fire of the opposition. I suppose such a force would have officers, non-commissioned officers, and soldiers of the line who all understood and were practiced in their roles in the company, regiment, division, and army. Poorly trained soldiers, especially those in the militia, would frequently break in combat and flee the battlefield. Thus, soldiers over a period of time were acculturated to roles in institutionalized military units, taking part in battle accordingly.
• Hamner describes the dangers faced by soldiers such as the Confederate soldiers taking part in Pickett's Charge in the Battle of Gettysburg. In that case, there were an equal number of Union soldiers on the ridges above, taking many fewer casualties while decimating the Confederates. Thus the Union and Confederate soldiers had very different experiences in the same engagement. However, before the battle, neither side would have known what the experience would be in that specific battle, and once in the battle it might be to late to back out.

What were the potential outcomes for a soldier in a Civil War battle?

• He might come out of the battle unscathed. Well that might be all right.
• He might be wounded, but not disabled. That might get him out of the worst of the war for some time -- perhaps not too bad an alternative. We know that soldiers have shot themselves to get out of the front lines.
• He might be wounded and somewhat disabled. That might get him to light duty and relative safety, or even a discharge.
• He might be wounded and seriously disabled.
• He might be killed.

Hamner pointed out in the lecture that hundreds of thousands of Union troops had enlisted for three years during the first year of the war. The government sought (desperately) to get them to reenlist, and offered them immediate one-month furloughs to go home and visit families if they did so. Many believed that they would not survive the rest of their original tour of duty, and reenlisted in order to get that home leave before they returned to duty and the death that they believed they could not avoid.

• For such a soldier, serious disability might be preferred to the death he believed was almost surely coming.
• For such a soldier, immediate death in battle might be preferred to death from disease in camp, death from disease in a prisoner of war camp, or death in a future battle after an extended period of discomfort, boredom and illness in camp between battles, punctuated by the horror of increasingly frequent battles and loss of friends and comrades.