icfis, leiden 21 august 2014 norman fenton queen mary university of london and agena ltd
DESCRIPTION
Limitations and opportunities of the likelihood ratio approach for evidence evaluation. ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd [email protected]. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/1.jpg)
ICFIS, Leiden
21 August 2014
Norman Fenton Queen Mary University of London and Agena Ltd
Limitations and opportunitiesof the likelihood ratio approach
for evidence evaluation
Fenton, N. E., D. Berger, D. Lagnado, M. Neil and A. Hsu, "When ‘neutral’ evidence still has probative value (with implications from the Barry George Case)", Science and Justice, Volume 54, Issue 4, Pages 274–287, July 2014
![Page 2: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/2.jpg)
Overview
1. Revisiting the Likelihood Ratio – first principles, theoretical benefits and limitations
2. Revisiting the LR in two well-known cases3. Why Bayesian networks are needed 4. Conclusions and way forward
![Page 3: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/3.jpg)
1. REVISITING THE LIKELIHOOD RATIO – FIRST PRINCIPLES
![Page 4: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/4.jpg)
Was Mrs Peacock the murderer?H: “Mrs Peacock guilty”
P(H) = 1/6
E: “The murderer was female
P(E | H) = 1
P(E) = 1/2
By Bayes P(H | E) = 1/3
![Page 5: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/5.jpg)
When does evidence E support a hypothesis H?
When our belief in H increases as a result of observing E, i.e. when P(H | E) > P(H)So, suppose E supports H and that H’ is an alternative mutually exclusive hypothesis. Can we conclude that our belief in H’ must have decreased? NO!! (suppose H’ is “Miss Scarlett is murderer”)Except when H’ = not H
P(H’ | E) = 1 – P(H | E) < 1 – P(H) = P(H’)
![Page 6: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/6.jpg)
A simple formal definition of probative value of evidence
The ratio R: • R > 1 means E supports H• R < 1 means E supports not H• R = 1 means E is neutral for H
Why do we never see this definition used?
![Page 7: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/7.jpg)
Because of obsessive and irrational fear of the explicit
PRIOR P(H)
![Page 8: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/8.jpg)
Instead the Likelihood Ratio is usedIn addition to H we have to consider an alternative
mutually exclusive hypothesis H’
L= By Bayes Theorem providing that H’=not H we can
conclude• L>1 if and only if R>1• L<1 if and only if R<1• L=1 if and only if R=1So L is a valid measure of the probative value of
evidence E for H (when H’ = not H)
![Page 9: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/9.jpg)
Benefits of the LRSimple formula for probative value of evidenceNo need to explicitly consider prior for HForces forensic experts to consider the likelihood of both the prosecution hypothesis and the defence hypothesis
![Page 10: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/10.jpg)
But note:It is meaningless to talk
about the Likelihood Ratio being a measure of the
probative value of evidence without explicit reference to
Bayes Theorem
![Page 11: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/11.jpg)
And especially note…..When
H’ ≠ not H
the notion that the LR is a measure of probative value of evidence is tenuous and potentially misleading
![Page 12: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/12.jpg)
When H’ ≠ not H…• Knowing that LR>1 just tells us that the ratio
of posterior probabilities (of H and H’) is greater than the ratio of prior probabilities
• So all we can conclude is E supports H more than it supports H’. But E may not support H at all because we can still have P(H|E) < P(H)
• Similarly LR=1 only tells us E supports both H and H’ equally. That does NOT mean E is neutral; P(H|E) might be very different to P(H)
![Page 13: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/13.jpg)
Was Mrs Peacock the murderer?H: “Mrs Peacock guilty”
E: “The murderer was female
P(E | H) = 1P(E | not H) = 2/5
LR= 2.5
But if H’: “Miss Scarlet guilty”P(E | H’) = 1
LR=1
![Page 14: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/14.jpg)
Issues and limitations of the LR• ‘probative value’ is not what people think it means
when H’ is different from not H• But it is difficult to work with exhaustive pairs of
hypotheses• Priors can never be truly ignored• Evidence E is rarely ‘simple’ – normally involves E1
and E2 that require separate likelihoods• Can be difficult even to avoid non-mutually exclusive
hypotheses in practice• Even if we get it all right LR of source level hypotheses
tells us NOTHING about LR of offense level hypotheses
![Page 15: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/15.jpg)
ExampleFred and Joe live at the same address. Gun X is registered to that address. Bob is found murdered from a gun shot. Evidence E: “there is a gun in Fred’s house with FDR that matched that from the crime scene.” Fred is charged with the murder of Bob. The offence level hypotheses are: Hp: Fred fired the shot that killed Bob not Hp: Fred did not fire the shot that killed Bob
The source level hypotheses are:Hp1: Fred owned gun that killed Bob not Hp1: Fred did not own gun that killed Bob
![Page 16: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/16.jpg)
Some reasonable assumptions
LR=1 for source level hypotheses…..but E has real probative value on Hp
Essentially irrelevant
![Page 17: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/17.jpg)
Prior state of the BN
![Page 18: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/18.jpg)
Calculating the probability of evidence E under the two values for H1p
P(E | not H1p) = 0.9891 (unchanged from prior)
P(E | H1p) = 0.9891 (unchanged from prior)
![Page 19: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/19.jpg)
Evidence is observed
Probability of Hp jumps from 1% to over 9%
..the evidence is certainly not neutral
![Page 20: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/20.jpg)
2. REVISITING THE LR IN TWO WELL-KNOWN CASES
![Page 21: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/21.jpg)
R v Sally Clark 1999-2003
Convicted and ultimately cleared of murdering her 2 children
![Page 22: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/22.jpg)
Sally Clark Revisited: A new issue in the probability experts’ reasoning
Hd : Sally Clark’s two babies died of SIDSHp : Sally Clark murdered her two babies
“(Prior) probability of Hd over 100 times greater than (prior) probability of Hp”“So assuming LR of 5 posterior of Hd still 20 greater
Hd : Sally Clark’s two babies died of SIDSHp : Sally Clark murdered at least one of her two babies.
(Prior) probability of Hd only 2.5 times greater than the (prior) probability of Hp
![Page 23: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/23.jpg)
R v Barry George, 2001-2007
Jill Dando
![Page 24: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/24.jpg)
R v Barry George (revisiting the Appeal Court judgment)
Hp: Hypothesis “BG was man who shot JD”E: “Single particle of FDR matching that from the gun that killed JD found in BG coat pocketDefence likelihood P(E|not Hp) = 1/100…But Prosecution likelihood P(E| Hp) = 1/100So LR = 1 and evidence ‘has no probative value’But the appeal transcript suggests a problem…
Fenton, N. E., D. Berger, D. Lagnado, M. Neil and A. Hsu, "When ‘neutral’ evidence still has probative value (with implications from the Barry George Case)", Science and Justice, Volume 54, Issue 4, Pages 274–287, July 2014
![Page 25: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/25.jpg)
Confusion from experts about the hypotheses
• Not clear that Hp stated was really the same prosecution hypothesis considered by the experts – H1p: “The particle found in BG’s pocket came from a gun
fired by BG”.– H2p: “The particle found in BG’s pocket came from the gun
that killed JD”.
• Transcript suggests the experts did not adhere to the assumption that defence hypothesis Hd was simply “not Hp”, i.e. “BG was not the man who shot JD”.– H1d: “Integrity of BG coat was corrupted”
![Page 26: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/26.jpg)
LR=1?
P(E | Hp) = P(E | H1d) but the evidence E is not neutral as concluded by expert and accepted by the court. It favours Hp.
![Page 27: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/27.jpg)
3. WHY BAYESIAN NETWORKS ARE NEEDED
![Page 28: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/28.jpg)
More comprehensive BN model needed in BG case
![Page 29: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/29.jpg)
Target is type X
Target is source
Source is type X
Target tested X
Source tested X
Even single piece of forensic match evidence is NOT a 2-node BN
Source is type X
![Page 30: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/30.jpg)
Bayesian nets: what we need to stress
Separate out assumptions from calculationsCan incorporate subjective, expert judgementCan address the standard resistance to using subjective probabilities by using ranges.Easily show results from different assumptions
…but must be seen as the ‘calculator’
![Page 31: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/31.jpg)
The potential of Bayesian Networks
“I assert that we now have a technology that is ready for use, not just by the scholars of evidence, but by trial lawyers.”
Edwards, W. (1991). "Influence Diagrams, Bayesian Imperialism, and the Collins case: an appeal to reason." Cardozo Law Review 13: 1025-107
![Page 32: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/32.jpg)
4. CONCLUSIONS AND WAY FORWARD
![Page 33: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/33.jpg)
Summary• LR and probative value of evidence may not
be what people think it is• In isolation the LR may be highly misleading• Doing things correctly requires fuller models
- BNs• But Bayesian arguments cannot be
presented from first principles
![Page 34: ICFIS, Leiden 21 August 2014 Norman Fenton Queen Mary University of London and Agena Ltd](https://reader037.vdocuments.us/reader037/viewer/2022103101/568143a6550346895db02a86/html5/thumbnails/34.jpg)
Blatant Plug for Book
CRC Press, ISBN: 9781439809105 , ISBN 10: 1439809100