Expert e-wire
  Sign me up!  
  Have your say!  

Not logged in
  Log in now  

Join up!
Benefits for experts
Application form
Apply online

Benefits for lawyers

Did we help?
  Feedback   
  Tell a friend   
  Contact us...   

Professional Indemnity Insurance for Expert Witnesses
Top quality
PI Insurance cover
at market-beating prices

Little Books
The Little Books
We have learnt the lessons from the mistakes of others, now you can learn them too!

Expert Witness
Year Book
The Expert Witness Year Book
Slip one in your bag, and you can be the expert with the facts at your fingertips!
  In all likelihood

The Court of Appeal looked recently at likelihood ratios and concluded that justice is not served by dressing up an expert's guesses as pseudo-science

In R -v- T 1, the Court of Appeal has looked again at the use by experts of mathematical formulae for calculating likelihood ratios. In this case, the appellant was tried for murder. The sole issue was that of identification.

Following conviction, he appealed on three issues. One concerned the extent to which evaluative evidence on footwear marks is reliable and the way in which it was put to the jury. This raised an issue of some importance in relation to the use of likelihood ratios in the provision of an evaluative opinion where the statistical data available are less than solid.

Mark of a marque

The facts in the original trial were these. A footwear expert from the Forensic Science Service (FSS) attended at the scene of a murder and recovered footwear marks. He carried out a comparison with a pair of Nike training shoes that were found at the appellant’s house following his arrest. In accordance with current practice in England and Wales, the expert carried out his examination based on a comparison of four factors: the sole pattern, the size, the degree of wear and the presence or otherwise of any damage. In his reports, he summarised the general approach that was taken to the evaluation of footwear mark evidence. He said:

‘... when assessing the significance of any correspondence or difference between a shoe and a mark resulting from a comparison, the likelihood of obtaining a correspondence or difference is considered against two alternative propositions: firstly, the shoe in question has made the mark it has been compared with, and secondly, that it has not made the mark. When the results of an examination are not conclusive the strength of assessment of the scientific evidence is normally expressed as a level of support for one or other of the above propositions. In attempting to convey the level of this support, the most appropriate expression is selected from among the following progressive verbal scale: very limited, limited, moderate, moderately strong, strong, very strong, extremely strong.’

The expert’s examination had revealed that the sole pattern and the size of the trainers were a match for the marks found at the scene. The sole pattern was, however, one of the most frequently encountered sole patterns on Nike trainers. The trainers were more worn than those that had made the marks, but the expert thought that additional wear might be explained by their use in the intervening period. He acknowledged that there was uncertainty about this because the amount of wear would depend on the extent to which they had been used. Furthermore, the marks at the scene indicated that there might have been damage to the trainers that had not been found on the pair recovered. A possible explanation for this might have been that the ‘damage’ marks had been the result of an artefact on the surface of the floor; that the area of damage had been worn away by subsequent use; or that the damage mark was caused by a stone lodged in the tread that had subsequently fallen away.

In relation to the correspondence found between the marks and the examined trainers, the expert considered it unlikely that these were attributable to coincidence. He concluded that there was ‘moderate scientific evidence’ to suggest that the appellant’s trainers had made the marks found at the scene.

A hidden formula

After the trial, the papers of the FSS scientist were examined by another expert and some of these showed the use of a formula to calculate a numerical value for the likelihood embodied in the progressive verbal scale. Although not referred to in the expert’s report, this likelihood ratio had been calculated according to a widely used formula - the Bayesian approach. It resulted in a numerical value that could be used to ascribe the data to one or other of the expressions in the progressive verbal scale. The formula used produces results that range between <10 (equating to weak or limited support for the hypothesis that a given shoe made the mark) up to >1,000,000 (extremely strong support). The number is a ratio of two probabilities, i.e. the probability that the proposition (that the shoe made the mark) is true divided by the probability that the alternative proposition is true.

In this case, the expert had applied this approach to each of the four indicators using the formula Pattern x Size x Wear x Damage with values of 5 x 10 x 2 x < 1 giving a likelihood ratio of ~100. This was given in evidence using the  verbal equivalent of ‘moderate support’. The values ascribed had been reached by the expert’s use of his own skill and judgment as well as by reference to statistics stored on a database recording information such as the numbers of particular makes and design of shoe sold over the course of a specific year.

Once it became apparent that the expert had used likelihood ratios, counsel for the appellant sought to appeal the conviction on the ground that the Court of Appeal had previously rejected Bayesian-type evidence in non-DNA cases.

On appeal

The Court of Appeal studied statements from principal scientists at the FSS, and also considered a note from the Forensic Science Regulator. The court, in its redacted judgment, referred to the paper Standards for the Formulation of Evaluative Forensic Science Expert Opinion, published in 2009, which contains the clearest statement of the approach adopted by the expert in this case. The paper defined ‘evaluative opinion’ as being:

‘... an opinion of evidential weight (evaluation of likelihood ratio) based upon case specific propositions and clear conditioning information (framework of circumstances) that is provided for use as evidence in court. An “evaluative opinion” is an opinion based upon the estimation of a likelihood ratio.’

The paper sets out the procedures to be adopted and what it terms ‘the guiding principles’ that justify this approach – the duty of the expert to base his opinion on the four principles of balance, logic, robustness and transparency.

It has been claimed that one of the advantages of the Bayesian approach is that it provides a safeguard against an expert understating or overstating the significance of scientific evidence. The court identified a difficulty in that this approach is not used by all forensic examiners. Some simply base their assessment on their own experience and ‘have scant, if any, regard to databases’. Unless the report refers specifically to the evaluative techniques and likelihood ratio calculations, it is unclear whether an expert has employed a Bayesian approach to his evidence or whether he has based it simply upon his own assessment.

The Tsar goes too far?

The Regulator offered the opinion that the methodology set out in the 2009 paper provides a structured approach to the provision of an evaluative opinion that would make it clear the factors that had been considered and how these factors had been evaluated. He advocated this for all expert forensic evidence based on an evaluative opinion, and it should not, he said, be limited only to certain types of expert evidence.

Counsel for the appellant argued that an examiner of footwear marks could assess how a mark could best be enhanced or recovered, prepare test impressions for comparison, assess the degree of match by pattern, size, wear and damage, and take into account the time elapsed since the mark had been made and the date of comparison. He did not accept, however, that such an expert could go on to interpret the degree of match and assign a specific level of probability. The expert’s evidence, he suggested, should be limited to a statement of whether the mark could or could not have been made by a particular shoe.

A distinction was made between shoe marks and the evaluative evidence in a facial mapping case. In the latter, there are distinguishing characteristics that can be applied to faces, but in the case of a shoe there are mostly just ‘class’ characteristics – features that are common to many thousands of shoes and will not readily differentiate one shoe from another of the same make and age. Without some identifying characteristics, e.g. a unique damage mark, the appellant’s counsel argued that the expert should not be permitted to give evaluative evidence at all.

The court felt this was going too far. It  said that an opinion that a shoe ‘could have made a mark’ was not the same as saying there was ‘moderate scientific support for the prosecution case’. The court said that in some circumstances an expert examiner could go no further than saying the mark could have been made. In that case, this was a phrase that would be a more precise statement of the evidence and one that would be better understood by the jury. However, in some cases the expert should be able to go further and give a more definitive evaluative opinion, even in cases where there were purely class characteristics.

So, the court turned its attention to whether it is permissible to use mathematical formulae and likelihood ratios based on statistics to arrive at that evaluative opinion in footwear mark cases. It rejected the Regulator‘s view that a similar approach is justified in all areas of forensic expertise, saying that ‘... each area requires a separate analysis because of the differences that there are in the nature of the underlying data.’

Data quality is key

In deciding this question, the Court considered the present state of knowledge and also looked at the practices elsewhere in Europe, the USA and Australia. It concluded that an approach based on mathematical calculations was only as good as the reliability of the data used.

In DNA cases the match probability is based on the reliability of a large and statistically sound database. In footwear cases, however, the underlying numbers aren’t accurately known.

For example, the sales figures for Nike trainers were based on Nike’s own figures. They did not take into account those sold by other distributors or the presence on the market of unknown quantities of counterfeits. Neither did the statistics take into account other factors such as the increased number of shoes in a certain area due to discounting by a particular retailer, local fashion, etc. Indeed, the FSS’s own shoe database contained information relating to 8,122 pairs of shoes, a tiny sample given that around 42 million pairs are sold in the UK each year. Clearly, unless the FSS database happened to be a perfectly selected representative sample of the total shoe dataset, the likelihood ratio calculated from the FSS database would be different from that calculated by using national figures. Furthermore, the FSS database was not routinely available to all examiners.

The Court held that there was no sufficiently reliable basis for an expert to be able to express an opinion based on the use of mathematical formulae. Any attempt to assess the probability that a given shoe could have made a particular mark based on figures relating to shoe distribution was inherently unreliable.

The Court quoted the judgment of Rose LJ in the case of R -v- Adams 2. Whilst Bayesian theory might be an appropriate tool for statisticians to establish a mathematical assessment of probability, the ‘apparently objective numerical figures used in the theorem may conceal the element of judgement on which it entirely depends’ and would be entirely inappropriate in jury trials. Juries would, he said, be plunged into inappropriate and unnecessary realms of theory and complexity, deflecting them from their proper task. Calculating a probability according to a likelihood ratio formula may well have the effect of elevating the opinion evidence in the eyes of a jury or a judge who might not fully understand the science behind it. The Court considered it clear that, outside the field of DNA, likelihood ratios should not be used unless there was a firm statistical base.

An expert, it seems, can give an opinion based upon a statistical database by simply using that database and expressing an opinion by reference to it (R -v- Abadom3). Similarly, he can make use of a scale to express an opinion, where this is permitted by the court.

In Atkins & Atkins -v- R 4, the Court approved the use of a scale to express an expert opinion. In that case, however, the scale was based solely on the judgment of the expert and did not involve any calculation based on a mathematical formula. There was no question in Atkins of using a likelihood ratio because there was no statistical database upon which to base any calculation.

Don’t give scientific credibility to guesswork

In allowing that an expert in a footwear mark case could give a more definite evaluative opinion (than the shoe could or could not have made the mark), the Court of Appeal said it was essential that the report made it clear that the expert’s view was subjective and based on his experience. It thought it best that the word ‘scientific’ should not be used in this context because it would give the jury an impression of precision and objectivity that is not currently present in this area of expertise.

The effect of the judgment appears to be that an expert can give an evaluative opinion based on a verbal scale of values. He can even refer to a statistical database, where such a database exists, and can make reference to it in his report. But he must make it clear in his report that the view is a subjective one. What he cannot do is make use of a Bayesian or similar mathematical formula in calculating a likelihood ratio unless there is a trusted and accepted database upon which to base such an analysis.

Currently, the courts have only agreed to this approach being used in cases involving DNA identification evidence.

Nowhere to hide

Given all this, what is to prevent an expert from making use of a Bayesian approach to arrive at his ‘subjective’ opinion and simply not referring to the fact in his report? The answer would lie in the Court’s insistence on transparency under Part 33 of the Criminal Procedure Rules. Where the mark could have been made by the footwear, the factors that enable the expert to express a more definite evaluative opinion must be set out, including any data that have been relied upon.

In R -v- T 1, the expert had justified his exclusion of the formula and statistics from his report by the explanation that these might have confused the jury. He said, in any event, that he had not relied only on the formula but had simply used it in accordance with the then emerging practice within the FSS as an aid to ‘standardising’ evaluative opinions across the board. This, said the Court, could be no justification because the Court must know what had been done.

The decision of the Court in Abadom explains the importance of referring to all material so that ‘... the cogency and probative value of the conclusions can be tested and evaluated by reference to it’.

The Court allowed the appeal and quashed the conviction on the grounds that it was unsafe. In doing so, it attached no blame to the expert in the case, saying he had merely followed what was then standard practice in the FSS. However, the Court found that the statistical evidence was inherently unreliable, that the basis on which the calculation had been made was not put before the jury, and that the process had not been transparent. Due to these factors, the defence had been unable to mount a proper challenge to the statistical evidence. It was clear upon hearing the new evidence on these statistics that the figures given to the jury based upon the limited FSS database had been more favourable to the prosecution case than would have been the case if national data had been used.

Supporters of Bayesian theory argue that there is nothing wrong with the technique but that jurors are confused by the way in which its use must be explained to them. It has been likened to explaining a simple mathematical sum on a calculator by reference to a sequence of binary functions through logic gates.

Some supporters argue that all the jury needs to know is that a value has been arrived at by a trusted formula for calculating probabilities. This is a hopeless over-simplification of the problem of likelihood ratios in expert evidence. If the courts were to allow the increased use of such formulae and extend it, albeit with appropriate warnings to the jury, to scientific areas where statistical records are incomplete or unreliable, or where statistical analysis is open to differences in interpretation or scientific disagreement, this would be to lend a degree of certainty to an expert opinion that was wholly unjustified. The Court of Appeal has got it right in this case. Justice is not served by dressing up an expert’s guesses as pseudo-science.

References

1 R -v- T [2010] EWCA 2439

2 R -v- Adams [1996] Crim LR 898

3 R -v- Abadom [1983] 1 All ER 364

4 Atkins & Atkins -v- R [2009] EWCA Crim 1876

Valued this article? or

 

 

 
Issue 70
July 2011

In all likelihood
Experts and the ultimate issue
Conference notice


Current issue
September 2017

New rules when suing individuals for fees
To whose benefit?
Promoting the hot tub
Conference news
Not logged in -  Log in now