Kurt Walberg writes:
I think it would be great if the media, or media watchers, had some tool that would rate reliability of an author or article in an objective, reviewable way. I’m aware of Newsguard, but it is so general/overviewish as to be virtually useless. Any chance that a statistical approach could pull that off? What would it look like?
My response: I don’t know but there is some work on prediction markets for science. Maybe this is the closest. But there’s a lot that it wouldn’t address. For example, are my writings trustworthy? I’ve made very few quantitative predictions. Lots of what I write is some combination of storytelling and logical argument, but there is a level of trust that makes it all work, a sense that I’m calling it as I see it and not trying to hide contrary evidence. I don’t see how this could be assessed through any sort of content analysis. Instead, you could assess my reliability could be assessed through some upvote/downvote-style social reputation mechanism, similar to yelp or whatever. But that could be gamed in so many ways.
Perhaps a better approach to this would be a standard ranking system and a central “rank bank” for science news stories, with a ranking tied to the contributing scientists. The public could rank the predictions in the story (“Studies Show Consuming 12 cups of Coffee A Day With One Lemon On the Side Increases Risk of Becoming President of Burkina Faso by 459%”) choosing from, say “Certain”, “Likely”, “Possible”, “Unlikely”, “Bullshit”. Each scientists’ contribution to the story would be cataloged and rated “Supports”,. “Indifferent”, or “Disputes” to the central claims in the story. Thus, if the central claim gets an overall rating of “Bullshit”, the scientists who dispute it would get a positive score and scientists who support it would get a negative score, and vice versa if the story gets an overall rating of “Certain”.
Yep. The scientist rating data bank. Now that would be some cool data to have a hack at.
Andrew: Maybe we’re approaching the whole peer review evaluation process the wrong way. How about a more economic approach to peer review that replaces detailed verbal/technical assessments of scientific merit, originality, innovativeness, contribution potential, etc., with olfactory evaluations?? A scratch-n-sniff approach to peer review that would give new—literal— meaning to expressions like “doesn’t pass the smell test”. Just encode each article with scratch-n-sniff scents chosen by the reviewers. Although the possibilities are endless, I’ll bet most of the heavy lifting could be done with just a few basic scents:
Plain Vanilla,
French vanilla
Fresh breeze
Stale
Fishy
Pungent
More scents:
Cotton candy for sugar-rush fluff pieces that have no real nutritional value.
Roast beef for substantial contributions on meaty topics
Roast turkey for articles that are deficient in so many ways that they otherwise defy olfactory classification