In this, I am following Richard Carrier in his book Proving History. I am persuaded by him that we ought not just to use adjectives like "possible" and "probable" when we debate which theories best explain the evidence before us on a contentious historical or forensic question. Additionally, we should use Bayes' Theorem: using numbers to express our opinions, and multiplying and dividing them according to Bayes' formula in order to calculate our reckoning of which theory explains the evidence the best. The three main virtues of Bayes' Theorem are that it forces the analyst of evidence to specify clearly how good they think a theory is at explaining the evidence; it enables them to put all the evidence together in a mathematically sound way; and, above all, it forces them to look for evidence that supports their theory better than alternative theories, thus helping them to overcome the common failure to give alternatives due consideration. Of course, different people can have different opinions about probabilities: the virtue of Bayes is that it brings out exactly what people agree and disagree about, and thus focuses their debate productively on crucial areas of disagreement.
Here is a simple account of how to use Bayes' Theorem:
- We shall test the evidence for theory T (i.e. Megrahi committed the crime) by comparison with theory Not-T or ~T (i.e. Megrahi did not commit the crime).
- Firstly, we determine the prior probability of T being true. This is the probability that T is true based simply on our background beliefs; that is, based on general knowledge we have before we start to consider the specific details of the case. E.g. in general, how probable is it that a bomb in a suitcase could be loaded at Malta? How probable is it that a bomb could evade detection while undergoing x-ray checks at Frankfurt and Hamburg?
- Next, we take each piece of specific evidence in turn and consider for each piece its conditional probability of existing if, firstly, T is true, and if, secondly, ~T is true. E.g. how probable is it that Megrahi would be present at Luqa airport on the day of the bombing if he were not there to load the bomb into the baggage system? How probable is it that a mysterious piece of unaccompanied baggage would have passed through Frankfurt airport's baggage transfer system, if it were not the bomb-case hypothetically loaded in Malta?
- For each of T and ~T, we multiply the probabilities of each piece of evidence existing, to produce the conditional probability of the whole set of evidence in existence, assuming the truth of each theory in turn.
- Finally, we calculate the product of the prior and conditional probabilities of the background and specific evidence, assuming the truth of T, and then assuming the truth of ~T, and then we put these two consequent probabilities into a ratio, giving us the relative probability of theory T. Since in this case T and ~T exhaust all the possible theories, we can multiply the consequent probability of T by 100 to produce a percentage.
If this does not make sense, please read try the explanations on these pages.
I want to perform a Bayesian analysis on the evidence that convicted Megrahi of the Lockerbie bombing, both out of interest to see the result, and in order to show up clearly which errors of reasoning, if any, the judges fell into.
Now, let's determine the prior probability that the narrative of the crime constructed in the verdict was a true reflection of events, considering only such general background knowledge as we have and was included in the verdict. An expert in airline logistics and criminology would obviously be able to give more expert opinions, but I will do the best I can according to my own lights. I will assume that if the bomb was loaded at Malta, then Megrahi was the guilty party. We can express the relative probabilities of each piece of evidence, background or specific, on each theory T and ~T as a ratio: 1:1 when the evidence is equally probable on either theory; 10:1 if it is 10 times more probable assuming the truth of T than ~T.
Background knowledge leading to a prior probability
- How probable was it that a bomb could be introduced onto a plane at Luqa airport?
The evidence of the responsible officials at the airport, particularly Wilfred Borg, the Air Malta general manager for ground operations at the time, was that it was impossible or highly unlikely that a bag could be introduced undetected at the check-in desks or in the baggage area, or by approaching the loaders, in view of the restricted areas in which the operations proceeded and the presence of Air Malta, Customs and military personnel. Mr Borg conceded that it might not be impossible that a bag could be introduced undetected but said that whether it was probable was another matter. ... the Crown accepted that they could not point to any specific route by which the primary suitcase could have been loaded ... The absence of any explanation of the method by which the primary suitcase might have been placed on board KM180 is a major difficulty for the Crown case... (p. 41-2)I would normally rate the probability of a bomb in a suitcase getting onto a plane at Luqa airport at lower than 1%, based on the expert evidence that it was "impossible or highly unlikely". I shall rely on this evidence, even though it could potentially be biased against the possibility by Mr Borg's care to safeguard his reputation. But, for the sake of making Megrahi's guilt as probable as we can reasonably make it, in order to find the maximum probability that he was guilty, I will go ahead and allow a 10% probability. This is way higher than the real probability, but if we use maximum probabilities then we will at least know by the final calculation that Megrahi's guilt cannot be any more probable than the number we calculate in the end.
- How probable was it that a bomb such as destroyed Pan Am 103 could have evaded x-ray detection at Frankfurt?
It was submitted that the x-ray would, in all probability, have detected any explosive device in a case, particularly as the staff at Frankfurt were aware of warnings to look out for explosive devices hidden in radio cassette players. One such warning was issued after the Autumn Leaves operation in October 1988. Another, more limited, warning was issued because there was understood to be a threat that a woman from Helsinki would attempt to smuggle a device on board an aircraft. It was submitted that that examination would have revealed the presence of the radio cassette player and its contents, particularly in view of the fact that there had been a warning to look out for explosive devices hidden in radio sets. (p. 37)However, with regard to Mr Maier, the relevant x-ray operator, whose written statements were entered as evidence since he was too ill to attend the trial:
Mr Maier explained that he had had some limited training in the use of the machine, but said that in the course of using it he had taught himself to distinguish various sorts of electrical equipment, and that he knew how to tell if explosives were present, from their appearance. Neither statement directly dealt with the question whether, and if so how, Mr Maier would detect explosives hidden in a radio cassette player. What he said was that the approach in dealing with electrical equipment was to see whether it presented a normal appearance, for example whether it had a plug. Other evidence, however, particularly the witness Oliver Koch, Alert’s trainee manager at the time, shows that the standard of training given to Alert employees was poor. That was also the view of the FAA investigators who visited Frankfurt in 1989. Mr Maier’s description of what he looked for does not suggest that he would necessarily have claimed to be able to detect explosives hidden in a radio cassette player. There was no expert evidence as to the ease or difficulty of detecting such hidden devices. The x-ray examination is one of the factors to be taken into account but it is only one factor to be weighed along with the others. (p. 38)It is difficult as a non-expert to know the probability of the bomb being detected. The existence of the warning argues in favour of it, but the low quality of training argues against it. It is frustrating that such a low standard of evidence was left unimproved due to the illness of a man. Surely courts should have a means of asking for important evidence that was not presented at trial before reaching a verdict! Anyway, I will allow an outside 50% chance that the bomb could have got through this dubious security system.
We need to multiply together the probabilities of the bombs being loaded at Luqa and evading detection at Frankfurt. 10% multiplied by 50% gives a 5% outside chance. So, based on these elements of background knowledge, it was at best highly improbable, 19-to-1 against, that the crime could have been committed as the verdict said.
It was a highly unlikely crime to come off. That does not mean Megrahi did not beat the odds and commit the crime in this way. If the specific evidence of his involvement is sufficiently strong, in that it demands decisively to be explained by his guilt as per theory T, then the conditional probability may be high enough to overcome the inherent improbability of the crime.
Specific evidence leading to a conditional probability
Here is the concluding paragraph of the verdict listing the main points of evidence adduced against Megrahi. I have coloured the separate points that I will deal with so as to bring out how I will deal categorise and consider the evidence:
We are aware that in relation to certain aspects of the case there are a number of uncertainties and qualifications. We are also aware that there is a danger that by selecting parts of the evidence which seem to fit together and ignoring parts which might not fit, it is possible to read into a mass of conflicting evidence a pattern or conclusion which is not really justified. However, having considered the whole evidence in the case, including the uncertainties and qualifications, and the submissions of counsel, we are satisfied that the evidence as to the purchase of clothing in Malta, the presence of that clothing in the primary suitcase, the transmission of an item of baggage from Malta to London, the identification of the first accused (albeit not absolute), his movements under a false name at or around the material time, and the other background circumstances such as his association with Mr Bollier and with members of the JSO or Libyan military who purchased MST-13 timers, does fit together to form a real and convincing pattern. There is nothing in the evidence which leaves us with any reasonable doubt as to the guilt of the first accused, and accordingly we find him guilty of the remaining charge in the Indictment as amended. (p. 82)I shall take the evidence against Megrahi point by point. Let us see if they really do "fit together to form a real and convincing pattern".
- The bomb-suitcase containing clothes from Gauci's shop in Malta, and Gauci's "identification of the first accused (albeit not absolute)" (there was a second accused who was found no guilty).
Mr Gauci picked out the first accused at an identification parade on 13 August 1999, using the words as written in the parade report “Not exactly the man I saw in the shop. Ten years ago I saw him, but the man who look a little bit like exactly is the number 5”. Number 5 in the parade was the first accused. He also identified him in Court, saying “He is the man on this side. He resembles him a lot”. (p. 55)Also:
Mr Gauci’s initial description to DCI Bell would not in a number of respects fit the first accused. (p. 65)Gauci actually mischaracterised Megrahi to the police in terms of his age and height.
Let us consider this evidence in Bayesian style. That means determining the probability that Gauci would say the purchaser of the clothes resembled Megrahi, and tell the police at first that the customer was of the wrong age and height, first assuming Megrahi's guilt, then assuming his innocence. If Gauci had indeed sold the clothes to Megrahi, I would expect him to pick him out fairly definitively. If Megrahi was not the man, then I would expect Gauci to say at best that Megrahi only looked "a little bit like exactly" the man and fail to give police an exact description.
What is bizarre is how the judges reasoned about this uncertain semi-identification:
Unlike many witnesses who express confidence in their identification when there is little justification for it, he was always careful to express any reservations he had and gave reasons why he thought that there was a resemblance. There are situations where a careful witness who will not commit himself beyond saying that there is a close resemblance can be regarded as more reliable and convincing in his identification than a witness who maintains that his identification is 100% certain. From his general demeanour and his approach to the difficult problem of identification, we formed the view that when he picked out the first accused at the identification parade and in Court, he was doing so not just because it was comparatively easy to do so but because he genuinely felt that he was correct in picking him out as having a close resemblance to the purchaser, and we did regard him as a careful witness who would not commit himself to an absolutely positive identification when a substantial period had elapsed. We accept of course that he never made what could be described as an absolutely positive identification, but having regard to the lapse of time it would have been surprising if he had been able to do so. (p. 66)That's right! The tentativeness of Gauci's identification was taken not as an indicator of its weakness, but explained away as care to be accurate! This is completely the wrong way to handle uncertain evidence. Gauci's identification should be held highly suspect because it was so uncertain, not lauded as the best that a careful witness would allow himself to say.
I would rate the probability of Gauci failing to identify Megrahi properly if he were the purchaser of the clothes as 50%. If Megrahi were not the purchaser, I would rate Gauci's semi-identification of Megrahi as perhaps 90%, given that on this theory Megrahi would merely have the misfortune of somewhat resembling the real purchaser.
So Gauci's semi-identification evidence is actually a blow for the theory of Megrahi's guilt by a ratio of 9:5. Percentage-wise, the probability of Megrahi's guilt, given Gauci's evidence, would be 5 in 14, thus 36%.
You can start to see what mistake the judges made: they chained together a series of improbabilities as if they added up to a probability. Rather, they multiply up to a greater improbability!
Note that Kerr in her book totally undermines the validity of Gauci's evidence. Among other things, Gauci had seen a photograph of Megrahi linking him with the Lockerbie bombing before he ever spoke with the police. Furthermore, Gauci also told police that a picture of an Egyptian terrorist named Abu Talb resembled the customer, and this suspect is known to have visited Malta before the bombing, giving him the opportunity to buy clothes at Gauci's shop.
- The transmission of an item of baggage from Malta to Heathrow.
An example of how Kerr describes the transfer-baggage mess:
Not only could no bag from Malta be discovered, at least thirteen more items remained unaccounted for. Given the sheer number of untraceable baggage trays there was really no reason to believe any particular one of them was the bomb, and the German police gave up. They appear to have been so thrown by the impenetrability of the worksheets they decided the whole thing was a complete waste of time. ... [The German investigator] doggedly hunted for some legitimate item of luggage that might have been in tray 8849 [supposedly transferred from the flight from Malta]. He failed – but then, he also failed to trace anything that might have been in at least five of the other trays on the printout, which puts that into a bit of perspective.Kerr quotes from the summary of the German police investigation of the Frankfurt element of the case:
This item of luggage cannot be said with certainty to originate from KM180 [from Malta] for the following reasons. [Many reasons given.] Throughout the inquiries into the baggage for PA103A there was no evidence that the bomb suitcase had been transferred with the luggage either from or via Frankfurt Main to London.She goes on:
The Scottish branch of the Lockerbie investigation, despite having found nothing on Malta to contradict that position, simply ignored him.So it is strange to read this from the judges:
The evidence in regard to what happened at Frankfurt Airport, although of crucial importance, is only part of the evidence in the case and has to be considered along with all the other evidence before a conclusion can be reached as to where the primary suitcase originated and how it reached PA103. It can, however, be said at this stage that if the Frankfurt evidence is considered entirely by itself and without reference to any other evidence, none of the points made by the defence seems to us to cast doubt on the inference from the documents and other evidence that an unaccompanied bag from KM180 was transferred to and loaded onto PA103A. (p. 38)The judges thought they had good documentary evidence that an unaccompanied bag transferred at Frankfurt from Malta. They did however consider some issues with the evidence, including the issue about an unaccompanied bag from Warsaw:
Reference was also made to another item in production 1060. ... this bears to show that an item coded at a station in HM at 1544 on 21 December also was sent to PA103A, and reference to the coders’ records bears to show that baggage from flight LH1071 from Warsaw was being encoded at that station at that time. It was agreed that no passenger from that flight transferred to PA103A, so that the records seem to show the presence of another unaccompanied bag on that flight. (p. 36)After mentioning this issue, however, the judges do not go on to provide any analysis of it. It is incredible! They had evidence that the supposed bomb-suitcase could have come equally well from Warsaw as Malta. Yet they mentioned this evidence only to ignore it completely!
If we include the evidence of a bag transferring from Malta, and say it was 10 times more likely that an unaccompanied bag transferred from Malta via Frankfurt on the hypothesis of guilt than on the hypothesis of innocence, then this would be strong evidence against Megrahi. So we can also say that the judges' misuse of the baggage evidence, if they were thinking in Bayesian terms, would have been a very significant mark against Megrahi (it would make my final calculation 72%, so positive but not really enough to exclude reasonable doubt). However, apart from Kerr's demonstration that the evidence did not necessarily mean what the judges thought it meant, the judges seriously misused even the evidence as they saw it. They completely ignored the evidence that the purportedly unaccompanied bomb-suitcase might have come as easily from Warsaw as Malta.
Let's see what would have happened if the judges had included in a Bayesian calculation the evidence about the Warsaw bag. This is actually an interesting bit of probability work. Assume that an unaccompanied bag has a high probability of containing a bomb, as is required by the judges' reasoning about the evidence of the bag that they thought came from Malta. What then is the probability that an unaccompanied bag would get onto the Frankfurt-Heathrow feeder flight from Warsaw on the hypothesis of Megrahi's guilt? Call that number x. It has to be a low number, since this bag cannot be the bomb, yet unaccompanied bags, ex hypothesi, are likely to contain bombs. It is therefore an improbable sort of bag. Now, what is the probability of such a bag feeding onto the flight from Warsaw on the hypothesis of Megrahi's innocence? Surely it is a number much greater than x. This is because, if an unaccompanied piece of luggage is regarded as likely to contain a bomb, and a bomb was fed onto the Heathrow flight, but that bomb did not come from Malta, then it must have come from Warsaw (or even somewhere else, since there were other dubious transferred bags). Therefore, assuming unaccompanied bags are likely to be bombs, which the judges must believe, the correct Bayesian interpretation of the existence of a piece of unaccompanied luggage feeding in from Warsaw is that this fact is much more likely on the hypothesis of Megrahi's innocence. Alternatively, if unaccompanied bags are regarded as not likely to contain bombs, then the evidence that an unaccompanied bag came from Malta is no more probable on the hypothesis of guilt that on innocence.
Since the judges included as a fact the existence of a suitcase from Malta, and might have rated that as, say, 10-to-1 evidence against Megrahi, by the same token they should really have rated the existence of a suitcase from Warsaw, equally likely to have existed, at the same 10-to-1 in Megrahi's favour. This is because, if the existence of a bag transferring from Malta makes it more likely that the bomb came from Malta, then the existence of a bag transferring from Warsaw does the same for the Warsaw origin. So the net effect of the evidence the judges had about luggage transferring at Frankfurt should have been neutral: 1:1
Kerr gets much closer to a correct understanding of the importance of the mysterious bag that seems to have come from Warsaw, even though no legitimate transfer baggage was supposed to come from that flight:
Both [the Malta and Warsaw mystery bags] were single items apparently coded at a time and a place where no legitimate luggage for PA103A was being handled. We’re completely out of legitimate luggage to reconcile to either of them. More lost, misrouted or rush-tagged cases being shuttled to Heathrow? Who knows. The fact is, though, that it has happened twice, and in the context of a significant number of other items being less than conclusively reconciled. Tray 8849 is anything but an isolated anomaly shrieking “bomb here!”
- Megrahi's association with Libyan intelligence and with people who sold and handled the MEBO timers, one of which set off the Lockerbie bomb.
It follows that Megrahi would have had the opportunity to commit the crime. But it does not follow that it is probable that he did it. There is a huge difference between possible and probable. Only if Megrahi's access to the timers makes it probable that he committed the crime should the judges draw a positive conclusion from this piece of evidence.
Let's look at this point in Bayesian style. What is the probability that Megrahi would turn out to have access to the MEBO timers if he committed the crime? Obviously 100%. What is the probability that Megrahi would turn out to have access to the MEBO timers if he did not commit the crime? Obviously still 100%! The access and association are certain on either theory because they refer to the time before the crime. What access and association Megrahi had before the crime is not evidence of whether he not he committed it, except in the sense that they made it possible for him to have committed it.
This was a crucial mathematical error by the judges. The (purported) fact of Megrahi's access to the right sort of timers only raises the probability that he committed the crime in the sense that it makes it possible rather than impossible. It does not make it probable, since even if he did not commit the crime, it would still be a fact that he had access to the timers. Therefore the correct use of this evidence is to regard it as indifferent towards the relative probabilities of theories T and ~T. It should have no positive effect moving the judges towards a guilty verdict, except in so far as making it possible that Megrahi committed the crime. There can be no movement above 50% probability however. The correct ratio to draw from his access to the timers is 1:1.
It is not as if only Megrahi had access to these specific timers, and it is not as if the Libyans could not have sold or supplied them to somebody else. The CIA had an equivalent that they had seized in Togo, so who knows what hands the Lockerbie one might have been through? It is even possible that the Libyan timers were used by Libyans to carry out the bombing, but not by loading the bomb at Malta. Warsaw, as we saw, was an equally probable origin as per the evidence the judges considered; and in fact Heathrow was a far more likely origin, as I explained in my previous post.
However, we can also ask how probable it was that the bombing would be perpetrated using these specific MEBO timers on the hypothesis that Megrahi did it (on orders from Libyan military or intelligence forces), compared with the probability on the hypothesis that he did not. It can be argued to be more likely that such timers would be used if Megrahi committed the crime than another, non-Libyan culprit. But since we could not say that the timers were not used by another Libyan in Warsaw or Heathrow, or supplied to other forces by the Libyans, I am not prepared to grant a significant imbalance in the probability in favour of Megrahi's guilt. There are lots of other possibilities that also involve the use of the same timer.
Furthermore, Kerr explains another alternative possibility:
The Blinkbonny fragment almost certainly originates from the same template as the Thüring boards [those supplied by MEBO]. That doesn’t, however, mean that it must have been made directly from that template. Printed circuit boards are easy to copy by photographing them, and any such second-generation copy would reproduce the tracking irregularities from the original. Anyone who had an original timer or even just a circuit board could make such a copy. ... had some other group who had got hold of one of the original models decided to copy this nifty little number for themselves?So the timer would not even necessarily be one supplied to Libya by MEBO; it could be a copy.
As I noted, Kerr's analysis makes this discussion of the timer academic, since she shows that it was not the same as that supplied to Libya after all. The timer's provenance is in fact rather mysterious.
- Megrahi's presence in Malta on a false passport at or around the material time.
Let's calculate the conditional probability of Megrahi's guilt, based on the specific evidence the judges relied on. The ratio of probability on T relative to ~T is 5:9 times 1:1 times 1:1 times 10:1 equals 50:9, or an 85% conditional probability of guilt. As can be easily seen, the only evidence that supports Megrahi's guilt is his failure to explain his presence in Malta on a false passport. The identification evidence is so weak as to work in favour of his innocence, the evidence of a transfer bag from Warsaw neutralises the evidence of a bag from Malta, and the timer evidence is indifferent between guilt and innocence. Megrahi should certainly have come up with a reason for his suspicious activity.
In order to find the probability of guilt that the judges should have arrived at by applying correct, Bayesian reasoning to the evidence they cited, we plug our numbers into the formula using Richard Carrier's online Bayesian calculator:
That is, the highest probability of guilt that the judges should have arrived at was 23%.
To explain the arithmetic, the prior probability of guilt (5%) is multiplied by the conditional probability of the specific evidence on the hypothesis of guilt (85%). The product is divided by the sum of the product of the prior probability (5%) and the conditional probability of guilt (85%) and the product of the prior probability of innocence (95%) and the conditional probability of the specific evidence on the hypothesis of innocence (15%):
(0.05 x 0.85) / (0.05 x 0.85)+(0.95 x 0.15) = 0.0475 / 0.185 = 23%.
Megrahi should have been found not-guilty (or alternatively not-proven in the Scottish legal system). This result is based on the evidence the judges had, except for the misused evidence about the baggage transfer which I excluded because its use contradicted the findings of the police report.
The crime was such an unlikely one that terrifically good specific evidence was needed just to make Megrahi's guilt turn out probable. If the conditional probabilities were 99:1 for Megrahi's guilt, then the consequent probability would turn out as 84%.
The judges' crucial error is clear and highly important as an example of how forensic and historical reasoning can go badly wrong:
They chained together a series of improbabilities as if they added up to a probability. Rather, they multiply up to a greater improbability!
The crime was hugely improbable on background knowledge alone. The identification evidence was so poor as to work in favour of acquittal, whereas the judges used it to help establish guilt. The evidence of Megrahi's access to the timer used was irrelevant, but the judges used it to help establish guilt. Of all the evidence, only Megrahi's failure to explain his suspicious behaviour was a real piece of evidence against him.
The judges failed to use Bayesian reasoning, which would have shown them that, far from a series of improbabilities adding up to a proof of Megrahi's guilt, they should have multiplied them out to a much greater sense of doubt. They failed to appreciate that the crime was such an unlikely one on principle, that iron-clad evidence of Megrahi's guilt was required to overcome the prior improbability: extraordinary claims require extraordinary evidence. A circumstantial case built on improbabilities does not cut it.
If the judges had applied correct probabilistic reasoning to the facts they did have about an unaccompanied bag from Warsaw, then this would have neutralised the evidence of an unaccompanied bag coming from Malta.
Of course, if we included the evidence explained yesterday, and considered how probable it was, on a hypothesis of Megrahi's guilt, that a mysterious suitcase answering to the description of the bomb-case would be seen by a baggage-handler at Heathrow before the feeder flight from Frankfurt had even arrived, then it would only be fair to divide the 23% we have come to here by maybe 10 times, if not more. Include all the evidence, and the probability of guilt is minimal.
Moreover, include a more realistic expectation of the probability of getting the bomb into the baggage system at Malta, and the probability drops again to a miniscule number.
In sum, even without the new understanding born of Kerr's investigation, Megrahi should not have been found guilty. With it, his innocence is proven.
Thus the worst mass-murder in British history, the killing of 270 people, should be regarded as an unsolved crime.
Thank you for reading if you got all this way!