Kevin Lewis points us to this article by H. Swofford, S. Cole, and V. King, which begins:
Over the past decade, with increasing scientific scrutiny on forensic reporting practices, there have been several efforts to introduce statistical thinking and probabilistic reasoning into forensic practice. These efforts have been met with mixed reactions—a common one being scepticism, or downright hostility, towards this objective. For probabilistic reasoning to be adopted in forensic practice, more than statistical knowledge will be necessary. Social scientific knowledge will be critical to effectively understand the sources of concern and barriers to implementation. This study reports the findings of a survey of forensic fingerprint examiners about reporting practices across the discipline and practitioners’ attitudes and characterizations of probabilistic reporting. Overall, despite its adoption by a small number of practitioners, community-wide adoption of probabilistic reporting in the friction ridge discipline faces challenges. We found that almost no respondents currently report probabilistically. Perhaps more surprisingly, most respondents who claimed to report probabilistically, in fact, do not. Furthermore, we found that two-thirds of respondents perceive probabilistic reporting as ‘inappropriate’—their most common concern being that defence attorneys would take advantage of uncertainty or that probabilistic reports would mislead, or be misunderstood by, other criminal justice system actors. If probabilistic reporting is to be adopted, much work is still needed to better educate practitioners on the importance and utility of probabilistic reasoning in order to facilitate a path towards improved reporting practices.
Interesting. Frustrating, but interesting.
The bit about defence attorneys taking advantage of uncertainty reminds me of a story from several years ago when I was doing some legal consulting. I prepared my report and then I had a conversation with the lawyer I was working with. It went like this:
Lawyer: So are you prepared to testify that the conclusions in that report are correct?
Me: Ummm, yeah, sure. I mean, it’s always possible I made a mistake somewhere, but I don’t think I made any mistakes—I was pretty careful—and if I didn’t make any mistakes, I stand by my claims.
Lawyer: You can’t say that!
Me: OK. What can I say?
Lawyer: You can say that your conclusions are correct.
Me: But . . . they could be wrong.
We went around on this a few more times and eventually came up with a phrasing that we were both happy with. Unfortunately, I can’t remember what that phrasing was. I’ve done legal consulting from time to time, and uncertainty has never been an issue. I get asked about the details of my reports; I’ve never been asked any vague questions like, “Could you possibly have made a mistake?” I suppose I’d respond to such a question with a generic response such as, “All things are possible, but I have no reason to believe I made any mistakes in that report, and I checked it over very carefully.” But I’m not sure.
Just to be clear, the above story is not intended to make me look good and the lawyer look bad. I’d say the opposite: the lawyer had a very reasonable request and I was being difficult. But I wasn’t being difficult for the sake of being difficult; I was being difficult because I was faced with a difficult challenge, which is how to convey near-certainty. After all, at some point we really can be essentially certain of things.
P.S. We most recently discussed criminology in our post, “You don’t want a criminal journal . . . you want a criminal journal,” in which a criminologist was quoted describing a request for data as a “blood sport.” If that’s what you think a blood sport is, I don’t think you’re a very serious criminologist.
P.P.S. The above-linked article by Swofford appeared in the journal Law, Probability and Risk. That sounded familiar, so I did a search of my papers and, yes, I’d published something there! It was my review of Nassim Taleb’s books. I just reread that review, and it’s great! I didn’t realize how many of my best stories I’d crammed into that short article.
I find this quite interesting and I expect similar results would be found with respect to medical providers, management consultants, and virtually any user of statistical analyses. Using the first of these as an example, many doctors (as exemplified in many standard practices that are derived from clinical trials) appear to resist probabilistic statements, often using the same concerns – people will misinterpret, patients want certainty, they will appear to be less of an expert, etc. Changing this will require a sea change in education, journalism, as well as the disciplinary training of people conducting statistical analysis (and the publication standards they live under).
However, my skepticism seems one-sided to me: there is a more profound reason for difficulty adopting probabilistic reasoning/reporting. We’ve encountered this when discussing election predictions. If I predict that candidate A has an 80% chance of winning, and then they lose, I was not wrong after all. There seems to be something unscientific about a conclusion that is never wrong. Perhaps it is because the probability only makes sense in a repeated setting while the prediction is for a single event. I think this mismatch is part of what respondents (like in the survey being discussed) mean by “misleading” or “misunderstood.” In fact, I think it is hard to express such results in a way that would not risk misleading or lead to misunderstanding while at the same time saying something of value to the consumers of research.
I’ll admit to having difficulty with my own portrayal of probabilistic results despite the fact that I am constantly doing so. I may know what I am saying when I provide my results, but I don’t think that is the same thing as understanding the needs of the consumers of my work. Sure, they want certainty, but even with uncertainty it isn’t so clear what they want or need. I try to provide a probability distribution over possible predictions, leaving it to their “judgement” to make their decisions. Am I meeting their needs? I’m not sure.
Once again it’s an issue with dichotomization, and frequentism. Is “80% chance of rain” wrong if it doesn’t rain today? No, it’s a poor model perhaps, but it’s only WRONG if it isn’t calculated correctly, like 1+1=3 is wrong. Because every probability is conditional on the assumptions…
p(rain | sensor_readings, model, calculations)
A probability follows from a set of assumptions. It can’t be wrong without a calculation mistake but the assumptions can be “wrong thinking” about the way the world works.
If you predict 90% chance Hillary wins and she loses by a small margin, most likely it was your model of the poll biases that was wrong. You can only conclude that by looking at the model, not the numerical probability
Daniel:
There’s also retrospectively wrong or prospectively wrong.
> Is “80% chance of rain” wrong if it doesn’t rain today? No, it’s a poor model perhaps, but it’s only WRONG if it isn’t calculated correctly, like 1+1=3 is wrong. Because every probability is conditional on the assumptions…
> If you predict 90% chance Hillary wins and she loses by a small margin, most likely it was your model of the poll biases that was wrong. You can only conclude that by looking at the model, not the numerical probability
I don’t get it. When Hillary doesn’t win (something that the model predicts that will happen with 10% probability) do you automatically conclude that it’s most likely than not that the model was wrong? (What’s the probability that the model is wrong? What’s the ex-ante probability that the model is wrong?)
Or most likely it was your model of the poll biases that was wrong only if she loses by a small margin? Naively it seems that if you’re going to make a difference it should be the other way around: if she loses by a small margin that’s closer to winning.
> (What’s the probability that the model is wrong? What’s the ex-ante probability that the model is wrong?)
I meant “What’s the probability that the model is wrong WHEN SHE WINS? What’s the ex-ante probability that the model is wrong?)
There are two questions:
1) is the probability consistent with the assumptions made in your model (ie. did your MCMC calculation or whatever converge and provide accurately calculated numbers). If it didn’t you can definitely say the probability itself was wrong.
2) Are the assumptions made in the model consistent broadly with reality. Obviously if a model predicts multiple things at once and all of them are in low probability regions of space, then hey the model is wrong. But if your MCMC worked then the probability you calculated from it was a correct probability… of a wrong model.
3) Sometimes people seem to think that there’s “a real probability out there in the world” and your model gave the wrong one. Like, the “real” probability that clinton wins was 50% and you predicted 90% and the problem was you didn’t get the right value… This is just not the case, and what I’m pushing back against. There is no “real” probability in the world for most things (some carefully constructed dice or Crypto RNG or galton apparatuses with verifiable stable frequencies aside). You can’t say that it was “wrong” because it wasn’t the “true” probability, you can say it was a bad model because it predicted a lot of stuff would happen which didn’t, but not that it got the “wrong” probability.
I would go further. There is no “right/true” or “wrong/false” in Bayes theorem. No matter how bad your model performs, if that is the only one considered then that is what you go with. The entire point is to compare the relative performance of various models (weighted by priors).
Likewise, there is no proof or disproof in science. That is due to affirming the consequent and Duhem-Quine thesis, respectively.
The whole subject of probability in the criminal justice system is very iffy. For one thing, imagine lawyers arguing about whose priors are better! For another, I’m sure that very few lawyers understand much about probability and statistics.
I remember working on my divorce decree with my lawyer and it had an inflation escalator clause for support payments, at my insistence. I could not understand his formulation, and I’m a physicist and engineer. To the extent I could understand, I didn’t think it was right. So I rewrote that section to be simple, clear, and correct, with a little procedure for computing the increases. But now, my lawyer couldn’t understand it!
Tom said, “The whole subject of probability in the criminal justice system is very iffy”
The whole subject of probability in lots of (maybe most) situations is very iffy – because probability is only used in situations that involve uncertainty — and “uncertainty” and “iffy” are pretty much synonymous.
Moreover, many people find uncertainty uncomfortable, so that enters into the problem as well.
Well put!
One of the axioms in P. Keoller’s article – The Lighter Side of Reference Points: the better you are at describing uncertainty, the worse you are at providing advice.
If you’re not clear on what probabilities are you probably shouldn’t use probabilistic language.
That sounds sensible until I reflect that no one really knows – or at least, there isn’t near-universal agreement on what probability is. I’ve always been in the counting camp – basically a frequentist viewpoint. With that view, it’s hard to deal with a once-only event. So one is forced to imagine that event as one of an imaginary collection of similar events. Is that worse than thinking about Newton’s first law – inertia – as being an unreachable idealization (because every real thing always has some forces acting on it so you can never really observe truly force-free behavior)? I don’t know.
At its root probability is combinatorics. Saying something is more probable means there are more ways for “the stars to align” (the collective known and unknown influences in combination) so that particular outcome occurs.
A frequency distribution is an indirect method of estimating this by assuming the next observations will result from the same processes, whatever they may be. But that is typically not the only information we have available.
Tom wrote:
“there isn’t near-universal agreement on what probability is”
I would phrase that differently. The problem is that probability can be five things before breakfast. The stochastic probabilities that determine energy dispersive x-ray spectra have little in common – epistemological, ontological, or aesthetic! – with win probabilities in horse races.
Can you expand on this? It sounds interesting but I don’t really see it. Eg, make a table comparing your two examples along the dimensions you mention.
https://en.wikipedia.org/wiki/Probability_axioms and possibly https://core.ac.uk/download/pdf/268083255.pdf ?
I think Statistics would be better off if it had been left as “a collection of calculations”.
Perhaps because I often encounter situations in which the person I’m assisting is having to make a binary Yes/No decision, they want me hear an almost black/white argument. They have hired me to do the thinking, and they want an answer. Wavering is interpreted as “No” in these cases. I am actually at peace with offering a close to black/white characterization – with the knowledge that I’ve analyzed the situation probabilistically, and would have made the recommended decision myself. I’d slip in some comment of the type “if something goes wrong, it would be this unexpected thing…” Just throwing my own experience in the mix here.
Separately, on the topic of forensics, I’d like to see rigorous analyses on the new digital technologies, such as eyeball scanning, voice prints, facial recognition. Can anyone point me to serious scientific studies of the accuracy of these technologies (with uncertainties well calibrated)? The mass media seem to just accept these fancy things as 100% accurate.
Another perspective I’ve encountered in teaching probability and statistics is a religious one: That probability and uncertainty are associated with subjects such as gambling that are frowned upon by some religions. This came up once in connection with a very bright student I had who got a job teaching mathematics at a Baptist high school. I turns out that there is a get-around often used in schools affiliated with some religions: Students use “spinners” to generate (what we would call) random numbers, that can then be used in illustrating some of the concepts in probability and statistics.
That is indeed how it goes and it has been that way for decades. The plaintiff lawyers in their Gulfstream jets don’t want the courts to question whether modern benzene/asbestos/etc. exposures might pose risks so low that the uncertainty around their causation experts’ opinions must be disclosed to juries, and the Brooks Brothers suit-wearing prosecutors don’t want their criminologists to be forced to disclose that the uncertainty around bite marks / DNA collection / etc. on which their opinions rest is somewhere between very, very wide and never measured (because it would certainly be laughably wide). And so forensic science at the courthouse, upon which liberties and great wealth depend, remains an embarrassment to Lady Justice.
Reading the comments, I want to emphasize a point I was originally trying to make. It is easy to blame the lawyers, the public, and decision makers for the resistance to accepting uncertainty and thinking in terms of probabilities. But I wonder if some of that fault lies with our analysis. If our predictions can never be “wrong,” or as Daniel suggests, it is our model/assumptions that are wrong (which they always are), then it isn’t obvious to me what value we are providing. I hope it is of value, since I do a lot of this kind of work myself, but there is a bit of a chasm between what I think of as good analysis and what decision makers may want. I’m inclined to say “that’s what they get the big bucks for” – that I give them the possibilities and some more quantitative sense of the likelihood of these, and their role is to take that information and decide. I suspect this is easier said than done, however.
Expressing near-certainty to someone who expects certainty in answers happens so many places, even in personal life. Husband was the first to accept my true answers to “Are we going to be together in 5 years?” and “How much do you value me?” despite my highly positive outlook. Past partners, not so much.
Bernease:
Yes, this reminds me of our discussion from a few years ago of the somewhat unstable way in which we use different sources of discourse in different aspects of life.