How Accurate Are DNA Identifications/

As I understand it, courts sometimes use DNA identification to incriminate or clear suspects. Material found at the scene of a crime is found to contain human cells, the DNA is identified, and then compared with the DNA of a suspect.

Popular sources seem to suggest that it is very, very unlikely that two people have identical DNA, or that the wrong person will be identified from a DNA test.

Lets say that the laboratory reports back that the suspect's DNA matches that of the sample from the scene. What does that actually mean?

According to Wikipedia:

Monozygotic twinning occurs in birthing at a rate of about three in every 1000 deliveries worldwide.

So, as I understand it, 0.3 percent of individuals in the population share identical DNA with an identical twin. Unless the existence of such an identical twin can be ruled out, the most one can say is that there in not more than one chance in 333 that someone else left that specific DNA trace at the scene of the crime.

Say that investigators try to rule out the possible existence of a monozygotic sibling. We know that mistakes happen in hospitals and that there are other ways that twins are separated at birth without their ever later learning of that occurrence. So, even if efforts have been made to assure that the individual in question does not have an identical sibling somewhere, that investigation should not be considered absolutely conclusive.

Add in the possibility of error in the lab -- mixing up the crime scene sample with that from another crime scene, mixing up the suspect's sample with that from another person, mistakes in reading the DNA test results, deliberate falsification of the findings. The confidence provided by DNA testimony goes down further.

How about the possibility that the DNA sample taken from the crime scene is not that of the person who committed the crime, but that of some other person? Is it not possible that that does occur? How would you estimate its probability?

How many judges or jurors would be able to really understand the testimony of a DNA witness, properly estimate the probability of a suspect being responsible for the crime from the DNA evidence alone, or able to combine the probabilities arising from the DNA evidence with that from other sources (eye witness testimony, circumstantial evidence, exculpatory evidence offered by the defendant, etc.?

I am not comfortable with the accuracy that the judicial process might achieve in cases involving DNA evidence. Still, such evidence obviously should influence verdicts if available. The question is how to be sure that it appropriately influences verdicts.

Is randomized case controled experiment always the best way to evaluate a treatment?

I recently read a report saying that an extract of a specific cultivar of the Cannabis plant was effective in treatment of Dravet Syndrome, a rare, severe form of intractable epilepsy. It is reported that children affected by this syndrome, suffering hundreds of seizures per month and responding to no other medication, become seizure free after taking the extract. I don't know if the media report is true, or if this is something which occurred in a number of cases or is an isolated event that might be explained by some other change in the child or the child's situation.

However, it suggests that perhaps something other than a standard randomized case, controlled, double blinded study would be appropriate for approval of the use of such an extract. A radical reduction of symptoms in a case for which there was no alternative effective treatment, especially for a totally debilitating or fatal disease, would seem to call for trials in other children, and as long as the treatment proved uniformly beneficial, why would one give a placebo to other afflicted children. I am assuming that in the case of Dravet Syndrome, the diagnosis is confirmed by detecting the genetic defect that causes the seizures, and that there is sufficient historical evidence from untreated clinical cases that the seizures continue.

Scientific Publications Change Credibility of Hypotheses

There is an article in the science section of the New York Times on the reproducibility of results published in scientific journals. I think that it is based on a misperception of the role of scientific publications.

Let me start with an example. Times of sunrise, sunset, high tide and low tide are very accurately measured and predicted. If a scientific journal were to publish an article on the measurement of such events, those results could be reproduced widely and easily. The science is well established, its credibility assured. Thus no scientific journal would now publish such results (unless of course, the rotation of the earth suddenly changed in an unexpected way).

So suppose a scientist comes up with a hypothesis that the sun rises not in the east, but in the west in a specific place on earth. That is a very surprising hypothesis, one that seems very likely, a priori, to be wrong. He goes there and measures the location of the sunrise for a year, and indeed his data show it to have risen in a westerly direction. He submits an article to a good journal; it establishes he is a credible person, and peer reviewers can find no fault in his methods, so his paper is published. Of course, his hypothesis is still judged to be wildly improbable, but others would seek to replicate his observations. If they did, an important lead would then be available for new discoveries for earth scientists. If, as would be likely, it was found that there was an abnormality in the magnetic field or that his instruments were flawed in a way he had failed to detect, and that the sun indeed rose in the east, that would be the end of it. The results would not be reproduced and confidence would be restored in the prior theory.

Scientific journals seek to publish interesting results, those which show evidence for something that was thought a priori to be unlikely. The most interesting experimental results are just those which most challenge the probabilities as estimated before the experiments. It seems reasonable that these anomalous results will not stand up; that published scientific research will be difficult to reproduce. something is unlikely to be true, as any scientist will tell you, an experiment that indicates it is true is likely to be faulty.

This would seem to be especially true as scientists explore things very far from common experience, where the experiments themselves become very difficult. Think of particle experiments at very high energies, superconductivity experiments at very low temperatures, astronomical observations at very long distances, etc.

An experiment that appears to challenge a well established idea, doing so by achieving very unusual and difficult to achieve experimental conditions, may be very suggestive of new ways to understand the world. Such an experiment would also seem likely to be an artifact

•  a set of observations that would occur only very improbably, but which by the laws of random chance occurred in a specific experiment
• a methodological error, difficult to detect, that fell upon a reputable researcher. 

That is why replication is important, and indeed why scientific integrity is important. Often the supporting or challenging evidence will be obtained by alternative experiments, done with different equipment, and done in other laboratories.

So irreproducible results are not indicative of something being wrong with science or scientific publication, but are a natural result of the use of scientific journals to direct attention at potentially important findings as the frontiers of science advance into more and more difficult experimental domains.

This incidentally is why the public should not rush off to demand doctors prescribe a new drug based on reading a newspaper report that it had very unexpectedly cured a condition in an experiment on rats. A single unlikely result leaves the theory still unlikely. Consumers should wait until the FDA, based on the advice of expert, experienced scientists judge that the evidence is then sufficiently credible of safety, efficacy and effectiveness to allow doctors to prescribe the medication.

Ben Goldacre: Battling Bad Science

"Every day there are news reports of new health advice, but how can you know if they're right? Doctor and epidemiologist Ben Goldacre shows us, at high speed, the ways evidence can be distorted, from the blindingly obvious nutrition claims to the very subtle tricks of the pharmaceutical industry."

There is a movement in international development to depend more on randomized trials for evaluation of development techniques/innovations. People who believe it is easy to do a good randomized trial, or that the results of randomized trials (especially as presented by those who would manipulate opinion) are to be ascribed high levels of confidence should watch this presentation a couple of times!

Thinking about causality

A lot of research, especially evaluation research, seems to infer causality from correlation. Of course we know that there are problems intrinsic to such inferences:

• correlation may be spurious, occurring as an accidental confluence of different variables
• two variable may be causally related by the first causing the second, the second causing the first, or each causing the other
• two variables may be correlated because some third variable causes both.

If one thing is seen as causing another, we may ask if it is a necessary cause, a sufficient cause, or both necessary and sufficient.

Of course, there are many other forms of causality. Think of a situation in which there are five causal factors, any three of which are sufficient but no two of which are sufficient. Or think of a large number of workers, each loading a container at his/her own rate. The container is filled when the total of all the contributions adds to the capacity of the container.

And/Or logic allows for very complicated causal structures, even when each factor is binary. An "and gate" provides a positive output if and only if all of its inputs are positive. An "or" gate provides a positive output if any of its inputs are positive. Very complicated propositions can be implemented through complex structures of such gates. Moreover, these are very restrictive kinds of binary logic concepts.

It seems to me that many evaluations assume that an observed correlation is not only demonstrative of causality, but of a specific form of causality. Let me give an  example. I recall a project in which an industrial extension program was used to improve the rate and success of industrial innovation. In one country it appeared to be very successful, with a large portion of the enterprises receiving assistance reporting success in their innovations. On the other hand, in another country the same approach to industrial extension proved very unsuccessful. It seemed that there were "hidden variables" such as the overall industrial climate. If the industrial climate was positive, an extension service would work, if not the extension program would not work.

Are We Alone?: Skeptic Check

I was listening to this program on the radio this morning. The topic was flu vaccine and the fears stirred by vaccines in segments of the general population. I was struck by a couple of things:

• The concept of "false balance". The media unfortunately sometimes tries to appear balanced by juxtaposing a scientific viewpoint, supported by evidence from carefully controlled experiments which have been replicated and described in peer reviewed publications with anecdotal evidence or worse with completely unsupported assertions. While such assertions may be well motivated by "true believers" with superstitious beliefs, they may also be meretricious assertions made for private gain.
• Our evolved interest in stories. People do like to tell and to hear stories. Often we can learn from the stories that are told. Check your newspaper and you will see how often information of general applicability is cloaked in the story of an individual or a family. The problem is that science advances by amassing data. The broadcast noted that "data is not the plural of anecdote".

Remember the old Skinner Box experiment in which an animal could get a food reward by pressing a lever. Often the animals would exhibit a complex chain of behavior before pushing the lever, apparently failing to recognize that only one element of that behavioral chain was needed. But the chain which does ultimately result in the benefit became "superstitious" in that sense that it was falsely believed to be necessary to the desired result/

It occurs to me that there is a fundamental difference in the two stories:

• I was immunized and I did not get sick.
• I was immunized and I got sick.

The event of "getting sick" makes a much more interesting story than the event of "not getting sick".

Your memory isn't what you think it is

There was an interesting segment (in four parts) on 60 Minutes about the reliability of eye witness testimony in criminal proceedings:

• Part I
• Part II
• Part III
• Part IV
  

The program made the point that juries tend to ascribe more credibility to eye witness testimony than it deserves. It also suggested that unless witnesses are handled very carefully, they too tend to give more credence to their memories of the crime than the memories actually deserve.

Of course, there is information in eye witness accounts, and consequently of course eye witnesses should testify to the police and to the courts. However, police should be well trained as to how to avoid biasing identifications. One possibility that was suggested is to have photo and live lineups handled by computers, with the interface carefully managed by professional psychologists to minimize bias.

It might also be useful to provide some training to jurors on the credibility of eye witnesses, in a standardized fashion.

I would extrapolate that we tend to believe eye witnesses too much in other circumstances, even when there are fewer constraints on their honest description of what they witnessed.

Randomized Trials in Development Economics

The Economist has an interesting article this week about the debate going on among practitioners of development evaluation over the importance of “randomised evaluations” in which different policies — to boost school attendance, say — are tested by randomly assigning them to different groups. There seems to be agreement that randomized evaluations are useful for the study of the effectiveness of specific interventions, but that they do not help much with the study of macro economic policies nor of institution building efforts. Indeed, there are many problems of finding representative situations in which to conduct such trials which are such that the results may be expected to predict the utility of the innovations when scaled up to a national or regional scale. The article sites a recent Brookings Institute meeting.

The Brookings Institute seminar titled "What Works in Development? Thinking Big and Thinking Small".

Doubts About the Expertise of Forensic Evidence

The television program, 60 Minutes, and the Washington Post are doing an investigation of a failure of the FBI labs, which for decades produced metallurgical results implicating bullets from victims as coming from the same lot as ammunition found in the possession of suspects. Turns out the analysis was not trustworthy, and the FBI has stopped using the techniques, but has not yet notified the people convicted based (in part) on "expert" testimony overstating the validity of the metallurgical evidence. Today's article in the WP describes a case in which the tainted metallurgical evidence was combined with questionable DNA evidence (possibly contaminated samples which were not cleaned before analysis), doubtful (and possibly misrepresented) evidence implicating the suspects revolver offered by a State employed witness who falsified his credentials, questionable blood evidence, and in which alibi evidence was not accepted.

Comment: It is unfortunate when too much reliance is placed in the judiciary process on the value of expert testimony. Non experts must consider the quality of the forensic science, and the credibility of the persons presenting that evidence.

Thus it is especially important that government agencies that are responsible for forensic science maintain the highest quality standards, and are punctilious in the tests that they authorize and the evidence they present. They should not only assure the professional qualifications of their staffs, but also have strong programs to assure the ethical conduct of staff, especially when offering testimony.

It is especially important that the criteria used in assessing the performance of forensic scientists and laboratories be well conceived. They should not be given credit for convictions, but rather for accuracy of results and scientific quality of testimony.

The prosecutors faced with a posteriori evidence that some of the evidence presented in past trials is tainted have a difficult task. They must decide whether the evidence was important in the jury's decision, and their incentives are to convict criminals.

The courts have the responsibility to make the system work. Judges should themselves be knowledgeable about the quality of forensic evidence in the cases that they try, should keep the advocates honest, should help keep the expert witnesses to present their evidence fairly, and should help juries to evaluate forensic evidence and the cridibility of expert witnesses well. JAD