OK, here’s an exam question for you:

Someone comes up to you and reminds you that three years ago he asked you for advice, you gave him advice, he followed it, and he’s been doing very well since. You’d like to conclude that your advice helped. Give three reasons why the data conveyed in this story do not provide strong evidence for this conclusion.

And here’s the story:

Four years ago, we gave some advice for a student who had some questions about research ethics in an organization where he’d worked.

I forgot about the whole thing entirely and then this correspondent sent me an update:

I [the former student] took a full time role with [a large health care company] and I work on using analytical tools to improve healthcare delivery and operations. I also decided to ditch a MBA [from a well-regarded university] – though they admitted me – in favor of a mostly online Data Science Master’s [from a different well-regarded university]. I couldn’t be happier, and I couldn’t be busier.

I want to thank you for your advice response in blog form. I’m not so nearly tormented now as I must have been, then. Perhaps you helped, but there’s no way to know without a control group.

And even with a control group, there’d be no way to know based on just one or two people. Also selection bias: we give out tons of advice and may be more (or less) likely to hear back when things turned out well.

The other interesting thing is that, if you follow the link and read that old post, you’ll see that I didn’t offer any direct advice at all! This is something it took me a long time to learn: you can usefully respond to a question without answering it directly, by just being open about what you know that might be relevant.