David Hogg points me to this discussion:
Martin Strasbourg and I [Hogg] discussed his project to detect new satellites of M31 in the PAndAS survey. He can construct a likelihood ratio (possibly even a marginalized likelihood ratio) at every position in the M31 imaging, between the best-fit satellite-plus-background model and the best nothing-plus-background model. He can make a two-dimensional map of these likelihood ratios and show a the histogram of them. Looking at this histogram, which has a tail to very large ratios, he asked me, where should I put my cut? That is, at what likelihood ratio does a candidate deserve follow-up? Here’s my unsatisfying answer:
To a statistician, the distribution of likelihood ratios is interesting and valuable to study. To an astronomer, it is uninteresting. You don’t want to know the distribution of likelihoods, you want to find satellites . . .
I wrote that I think this makes sense and that it would actualy be an interesting and useful research project to formalize this as a decision problem. I’ve seen this sort of question arise in genetics (where should the p-value threshold be when you’re selecting N out of a million genes) but it’s frustrating because the cost-benefit calculations always seem implicit. I’d like to see it done out in the open.
Hogg replied:
I agree! In one—just one—of my papers we put an explicit utility model: http://arxiv.org/abs/0910.2233
The utility model is on page 17 and we use it explicitly on page 18 and on. It is a web system (up now at http://nova.astrometry.net/ and running inside flickr.com ) that has to *decide* whether to return results to the user or not, given probabilistic information about a submitted image.
Cool.
that should be “Martin (Strasbourg)” meaning “Nicolas Martin of Strasbourg”
Hmm, this seems related to a paper I’m currently involved in writing up – though in this case our problem is not structured as a decision problem, because empirical confirmation would not be straightforward:
We calculate, for each site in a genetic sequence, the posterior probability of it being interesting (where “interesting” is well defined but not relevant to this discussion). We report a list of all sites for which the posteriors are above an arbitrarily chosen threshold of 90%. The question is how to interpret these results and how/whether to correct for multiple testing. What we decided to report is the expected number of false positives in the list, with a 95% credibility interval. (E.g.: The following 49 sites have posterior probability >90%; of these, 3 are expected to be false positives (95% interval: 1-6).) This is easily calculated when you have posterior probabilites available.
Comments?