How do I form my attitudes about scientific questions?

The lively discussion on Phil’s entries on global warming here and here prompted me to think about the sources of my own attitudes toward this and other scientific issues.

For the climate change question, I’m well situated to have an informed opinion: I have a degree in physics, two of my closest friends have studied the topic pretty carefully, and I’ve worked on a couple related research projects, one involving global climate models and one involving tree ring data.

In our climate modeling project we were trying to combine different temperature forecasts on a scale in which Africa was represented by about 600 grid boxes. No matter how we combined these precipitation models, we couldn’t get any useful forceasts out of them. Also, I did some finite-element analysis many years ago as part of a research project on the superheating of silicon crystals (for more details of the project, you can go to my published research papers and scroll way, way, way down). We were doing analysis on a four-inch wafer, and even that was tricky, so I’m not surprised that you’ll have serious problems trying to model the climate in this way. As for the tree-ring analysis, I’m learning more about this now–we’re just at the beginning of a three-year NSF-funded project–but, so far, it seems like one of those statistical problems that’s easy to state but hard to solve, involving a sort of multilevel modeling of splines that’s never been done before. It’s tricky stuff, and I can well believe that previous analyses will need to be seriously revised.

Notwithstanding my credentials in this area, I actually take my actual opinions on climate change directly from Phil: he’s more qualified to have an opinion on this than I am–unlike me, he’s remained in physics–and he’s put some time into reading up and thinking about the issues. He’s also a bit of an outsider, in that he doesn’t do climate change research himself. And if I have any questions about what Phil says, I can run it by Upmanu–a water-resources expert–and see what he thinks.

What if you don’t know any experts personally?

It helps to have experts who are personal friends. Steven Levitt has been criticized for not talking over some of his climate-change speculations with climate expert Raymond Pierrehumbert at the University of Chicago (who helpfully supplied a map showing how Levitt could get to his office), but I can almost sort-of understand why Levitt didn’t do this. It’s not so easy to understand what a subject-matter expert is saying–there really are language barriers, and if the expert is not a personal friend, communication can be difficult. It’s not enough to simply be at the same university, and perhaps Levitt realized this.

My friend Seth has a similar problem but even more so; he’s reduced to supporting his opinions from ideologically-minded journalists (see here and here). This is not to say he’s wrong on the substance–and I recognize that Seth’s attitudes on this topic come from diverse sources, not just one or two news articles–it still looks to me that he’s flailing around, grabbing all sorts of talking points off the web, without having the opportunity to talk with an actual expert. (I should also add that Seth’s primary interest here is not the climate science or even the policy issues–as Seth has written more than once, he supports restrictions on carbon emissions in any case–but in the more general issue of how scientists and others can be fooled into following a consensus.)

In any case, it’s not easy to evaluate expert arguments if you don’t have the right connections. For example, I don’t know what to think about macroeconomics. The economists I know do micro and, for that matter, I actually feel the ability to evaluate arguments in microeconomics all by myself (as regular readers of this blog may be aware). Macro is tougher, though: it confused me in 11th grade econ class and it’s confused me ever since. Sure, Paul Krugman’s an expert and has strong views, but I don’t know him. If Paul Krugman were my good friend, I’d probably be relying on his opinions here. By this I don’t mean that, just because someone’s my friend, that I trust him. Rather, when talking with close friends, I know enough about how they think that I can evaluate their arguments and get a sense of where they know more and where they know less.

Unfortunately, this mode of thinking isn’t “scalable,” as the expression goes. There aren’t enough experts around for everyone to have an expert as a personal friend. Even a person as well-connected as myself doesn’t know any experts in macroeconomics, and even someone as well-connected as Steven Levitt doesn’t know any experts in climate science, so what hope do the rest of youall have. I’d like to hope that you all think of me as your personal friend and trust everything I write on statistics, but, from your point of view, I guess that doesn’t make much sense!

Where have I disagreed with the scientific consensus?

Phil in his blog entries mentioned some National Academy of Science panel, and more generally, I think it makes sense to respect expert consensus, especially if, as with Phil, you have a sense of where the expert consensus is coming from. For example, when Seth links to a news report of some scientist saying that sea levels are falling, and the expert consensus says the opposite, I’m inclined to go with the experts.

But I think it would help to round this out with some discussion of areas where I’ve opposed the expert consensus, situations where I’m pretty sure that I’m right and the consensus was wrong.

For example, consider Bayesian statistics, which nowadays is standard if not hegemonic in many areas of statistical application and has a high enough status outside the world of statistics that it is often loosely used as a synonym for “rational.” It didn’t used to be so. When we came out with Bayesian Data Analysis in 1995, there was only one applied Bayesian textbook out there (Box and Tiao’s, from 1973, which had a pretty limited range of applications), and even as late as 1997, you had respected statistician Leo Breiman writing that “when big, real, tough problems need to be solved, there are no Bayesians.” Breiman was wrong even then–even setting aside Laplace and restricting oneself to the modern era, he was ignoring a couple of decades of hierarchical Bayesian work in application areas ranging from education to toxicology–but he was going with the scientific consensus. Or, to be more precise, the local scientific consensus of where he worked (the UC Berkeley statistics department) and the general scientific consensus of the statistics profession in his formative years.

Unfortunately, Leo Breiman didn’t have a “Phil” to educate him about Bayesian statistics. He worked in the same department as me, but he was on the 4th floor and I was on the 3rd floor, and we rarely saw each other–unfortunately, that’s just the way things were around there. Breiman was not a specialist in Bayesian methods or the foundations of statistics, and unfortunately the “Phil” figures he did rely on in this area were not well informed themselves.

Here’s the point, though: it’s not just that Breiman was wrong in a way consistent with the scientific consensus of his era; it’s that he was wrong in a way that was characteristic of being part of the consensus. Look at it this way: Suppose Breiman had had the same views about applied Bayesian statistics but without this consensus behind him. Then, at the very least, he would’ve been a bit more careful in his pronouncement, or the journal editor would’ve been a bit more careful about publishing his article. Suppose, to use Phil’s example, that a prominent biologist were to write that “when big, real, tough problems need to be solved, there are no Darwinists.” The author would need to defend his statement (and, no, keyword search on the word “data” is not enough), and the journal editor would do a double-take. But, given the scientific consensus, it was considered ok to say this. It’s not that the scientific consensus is stupid, it’s that some statements are so stupid that they only come because the speaker has processed some aspect of the consensus in a particularly ugly undigested form.

Now, I don’t want to overstate this. The scientific consensus in Leo Breiman’s mind in 1997 wasn’t much of a consensus at all. Bayesian statistics were everwhere, and the leading journal of applied statistics had already been edited more than once by Bayesians. Breiman was fighting a rearguard action. Nonetheless, he spoke in the voice of a consensus that once had been, and by shutting down his filters, it did not serve him well

I’ve also done battle with smaller consensuses, with one clear example (for me) being the consensus in the statistical literature that it makes sense to judge model fit using the distribution of discrete contingency tables conditioning on their margin. This is a difficult enough problem in high dimensions that it has become somewhat of a classic in Monte Carlo computation, but I think the underlying question makes no sense. To me, it’s another case where the existence of the consensus has switched off people’s brains.

From the other direction, coming from the outside, I tend to trust scientific consensuses when they are available. For example, I know little about biology, but I don’t see much need to listen to the supporters of Lysenkoism, or creationism, or whatever–what I have seen along those lines doesn’t seem very convincing. And, despite what the conspiracy theorist say, I think that an academic biologist with real evidence in favor of such offbeat ideas could go far these days. (One can distinguish between different sorts of consensus. In biology, my impression is that there is a consensus about evolution by natural selection and genetic drift, there is vigorous debate about speciation, and then there are various fringe ideas that are not so much rejected by the scientific consensus as living alongside the consensus, in different ways. I’m thinking here, for example, about various offbeat ideas on sex ratios and ESP that have been discussed now and again on this blog. There’s no biological reason these things can’t be true in some way, but they haven’t been demonstrated very convincingly, and they live in little worlds of their own.)

I imagine that if I were not a statistician, I’d have consensus views about statistics that were wrong, in the way that some of the received wisdom of consensus is wrong. Much as I feel uncomfortable with the conventional misconceptions about statistics held by non-statisticians (including people such as physicists, chemists, etc., who in some way should “know better”), I don’t see much room for escape from holding conventional views about other sciences that I have not been able to seriously investigate.

Summary

I don’t really have any strong conclusions here. On climate science, I can take the lead from the experts whom I know. On topics such as macroeconomics, I’m in the same boat as Seth and have little choice beyond going with the popular or semi-popular media. Most statistics questions I feel pretty confident in judging myself–although if causal inference is involved, I like to run things by Jennifer before expressing a firm opinion. When it comes to political science, I can ask the experts down the hall.

What do I recommend youall do? On subjects where Phil and I are the experts, I suggest you listen to what we have to say. Beyond that, I dunno.

35 thoughts on “How do I form my attitudes about scientific questions?

  1. Good grief! You write:

    My friend Seth has a similar problem but even more so; he's reduced to drawing his opinions from ideologically-minded journalists (see here and here). This is not to say he's wrong on the substance, just that he's flailing around, grabbing all sorts of talking points off the web, without having the opportunity to talk with an actual expert.

    This is absurd. My opinions aren't drawn from "journalists" — please see the post I wrote a few weeks before Climategate called "Three Things Elizabeth Kolbert Doesn't Know." They are three things no journalist knows. You won't find those points in any piece of journalism.

    Andrew, I wrote a whole paper about how modelers fool themselves. Climate modelers and the journalists who trust them are fooling themselves — I can see this clearly having taken several years to write that paper. This has nothing to do with flailing about or taking my talking points from the web. Let me repeat: Good grief.

  2. You wrote: On topics such as microeconomics, I'm in the same boat as Seth.

    Based on what you wrote above this, don't you mean "macroeconomics"?

  3. Seth: Sorry; you're right. I rewrote in an attempt to be more accurate and fair.

    Marcel: Typo fixed; thanks.

    I guess it's true that, just as journalism is the first draft of history, blogging is the first draft of journalism!

  4. I think you've raised two separate issues. One is the process by which consensus builds, entrenches, shifts, etc. The other is how rational people make rational decisions about information.

    It's interesting to me how in a few notable areas the two are lumped together: the idea that biologists are maintaining some (evil) consensus in favor of evolution and that climate scientists, etc. are doing the same with regard to climate change.

    One difficulty is that nutty people also latch on to experts. I know many people who believe what Limbaugh or Beck say; these are their trusted reporters and thus the experts at filtering information which these people see as truthful and valuable. And I can't tell you how many people I've met who recite hideous science regarding vaccines and autism. In other words, self-reinforcement works for cranks and fools, the biased and the deluded just as much. I've met numbers of people who sincerely believe Vietnam is holding US soldiers prisoner nearly 40 years on and they not only believe but list "evidence."

    My personal belief is that evolution and climate are particularly ripe for skeptics because the ordinary person has a general idea of what is meant. Try explaining the Standard Model. Everyone can tell the weather by looking outside just as everyone can look at the variety of plants and animals. The more accessible the bare evidence, the more we should find the impossible to convince. While there are cranks who attack relativity, they play to a small audience, but anyone can interpret general information about the weather or the existence of bears and that opens the field to all the human players. Have a religious perspective? You can fit that in. Unable to understand the actual science? There's someone who can tell you what it means in a way that fits what you want to believe. You can't verify data about ice cores in Greenland but you can walk outside on a cold day and say, "Doesn't feel like global warming to me."

  5. As a general proposition, figuring out who to believe really is tricky. I think people should give substantial weight to scientific consensus, but you don't want to base your whole belief system on that if you can help it: after all, the scientists of the day were in consensus that "continental drift" was nonsense; that ulcers were not caused by bacteria…funny, I wanted to give at least one more example — the journalistic "rule of three" — but couldn't think of one off-hand, so I did a web search for [scientific consensus wrong examples] and came up with a Wikipedia entry that gives 5 examples…and two of them are the ones I had already. I had expected dozens. I'm sure there are dozens, or, hell, hundreds, if you look at issues with much narrower implications.

    Anyway, there are indeed examples of wide-spread scientific belief being proven wrong by a small but committed group of creative scientists. Sometimes these scientists have been heroic, continuing to make their case in the face of ridicule, inability to get funding for their work, and so on. (I say "sometimes" rather than "often" because the only actual cases I know are the few on that wikipedia page, plus the now-accepted theory that meteor strikes have caused some past mass extinctions. And not all of these fit the "heroic loner" model). If someone says "scientific consensus is usually wrong", well, they're wrong about that. If they say "scientific consensus is sometimes wrong," fair enough, but "sometimes" covers a lot of ground: do they mean 1% of the time, 10% of the time, or 49% of the time?

    I get the impression that some people think that if a small group of scientists faces ridicule by mainstream scientists, the small group is probably right! But in fact, the small group is probably saying something ridiculous. (In the very recent past, and maybe still, there have been a few scientists — with actual PhDs! — arguing that HIV doesn't cause AIDS, for example).

    I think it's good if people keep in mind that scientific consensus can be wrong. But it's disappointing how many people will grasp at extremely implausible arguments. If you want to choose to believe the scientists on the fringe rather than the scientists in the mainstream, surely you should do that in part on the basis of whether what the fringe people are saying makes any sense! And sometimes you can judge that yourself, you don't need any damn panel of experts telling you what to think.

    When it comes to anthropogenic climate change, if someone wants to allocate some probability to the chance that the skeptics have it right, I think that's a very reasonable thing to do. Make it 90% mainstream, 10% skeptics, or even 75% mainstream, 25% skeptics if you are are heavily inclined towards the skeptical camp. But there are people out there who are 90-10 the other way! If you are an expert climate modeler and you think your colleagues have the science wrong, that's one thing. If you're just some schmoe who only knows what he reads in the papers, and you choose to assign a 90% or 95% probability to the conclusions of the small band of skeptics…where does that come from? Do you really think the experts in a field get it wrong 90% or 95% of the time?

  6. In 1958 Francis Crick proposed his "central dogma of molecular biology," which became the expert consensus in that field. Later radiation biologist Tikvah Alper and the mathematician John Stanley Griffith advanced the hypothesis that scrapie and Creutzfeldt–Jakob disease are were caused by infectious protein, seemingly contradicting the central dogma. Years later Stanley Prusiner and his team at UCSF isolated the mysterious protein which he called a "prion." This work won him a Nobel Prize. However it was very rough going for Prusner precisely because he went against the received wisdom. Here the "expert consensus" certainly worked against scientific advancement. Of course this and many other exceptions don't prove that one should disregard the experts– in general they are the better bet. But that's a better bet against crackpots and amateurs. A serious challenge deserves attention, not ridicule along with a smear campaign. Unfortunately the global warming alarmists have done just that and gotten away with it until Climategate.

    The emails and documents from CRU call into question the integrity much of the data used by the experts. The process by which instrumental temperature readings have been "adjusted" to get "homogeneity" has been anything but transparent. How can we trust the experts when their data might be fraudulent? How can we trust experts who won't show us their work? Do we really want to proceed to change the economy of the world without checking the codes? How can we check those codes if the authors won't release them?

    There's another important aspect to the global warming debate: politics. It's no secret that left of center people tend to support AGW and right of center people don't, although there are exceptions like Lomborg. Academia and the media are generally left of center, and it should be no surprise that they are part of the cheering chorus for AGW. To make matters worse, it can be a career killer in some places to raise questions about AGW. I have witnessed this personally. It can get very ugly, and we should not be surprised to find that many critical of AGW are retirees– people safe from reprisal. Most people are cowards and have mortgage payments to meet. For example at the national laboratories you keep quiet about your skepticism or your raise will suffer at a minimum. Criticism is a threat to funding and funding trumps scientific integrity.

  7. Phil,
    In the 2009 University of Illinois at Chicago survey of 10,257 earth scientists, with a 30% response rate, 82% of respondents attributed warming to human activity. Let's say they all have a 90% probability like you asserted for yourself. Weighted, with no regard for the skeptics, wouldn't I, a non-scientist, have to go with a 74% (0.82*0.9)?

    survey here: http://tigger.uic.edu/~pdoran/012009_Doran_final….

  8. > It's not that the scientific consensus is stupid, it's that some statements are so stupid that they only come because the speaker has processed some aspect of the consensus in a particularly ugly undigested form.
    > …
    > … To me, it's another case where the existence of the consensus has switched off people's brains.

    This point is valid, and well put, but in the conclusion, I think Andrew Gelman is being far too pessimistic.

    > "What do I recommend you all do? On subjects where Phil and I are the experts, I suggest you listen to what we have to say. Beyond that, I dunno."

    This is very pessimistic and skeptical of considered consensus, and contradicted by Andrew Gelman's daily life. Before I step into a subway train, I don't form opinions about the quality of considered consensus of civil engineers, and Mr. Gelman does not either.

    Commenter "jonathan" makes the point:

    > I think you've raised two separate issues. One is the process by which consensus builds, entrenches, shifts, etc. The other is how rational people make rational decisions about information.
    > It's interesting to me how in a few notable areas the two are lumped together: the idea that biologists are maintaining some (evil) consensus in favor of evolution and that climate scientists, etc. are doing the same with regard to climate change.
    > …

    If you step back and compare "Skepticism of Human Activity Causing Global-Warming/Climate-Change" to established cases of motivated obscurantism, like denying evolution and natural selection, and tobacco carcinogenicity, and the Jewish Holocaust of WWII, and the efficacy of the polio vaccine, and perhaps less established cases of motivated obscurantism like controlled demolition taking down the Twin Towers and HIV/AIDS, you see familiar patterns and similar techniques and motivations both sinister and innocent-by-way-of-ignorance/gullibility. It will seem like bad form to the self-described "Skeptics", but they could bring doubters into their fold by work – the work of authoritatively publishing their opposing immutable thesis, and welcoming that to be subjected to the highest standard of scrutiny. And what are we to make of the "Skeptics" doing everything _except_ that work?

    The considered consensus of the scientific experts, here, is slowly growing and publishing an opposing authoritative immutable thesis – far too slowly and too messily and with too much initial unwarranted speculation for an impatient world – but at least they are building something up for possible future champion to knock down. And if it resists being knocked down – we have a consensus where it would be "perverse to withhold provisional consent", using Sagan's phrase.

    As for motivation within this possible case of motivated obscurantism, how can I discount the astroturf and sympathetic goodwill David Koch has purchased and does purchase?

    If you draw the boundary of consideration small enough "I dunno" seems like honest skepticism of considered consensus. But what is the compelling reason to draw the boundary of consideration so small as to ignore case for motivated obscurantism?

  9. Phil: Do you really think the experts in a field get it wrong 90% or 95% of the time?

    No, I'm sure they're right much more often than that. But that's just an average. You have to look at the particular case. If the issue has large political implications, you should immediately discount the likelihood that the consensus is correct (assuming, of course, that you're just taking their word for it). If the behaviour of the scientists departs much more from the ideal than is typical of non-politicized fields, then you should discount it even more.

    You might consider, for example, the field of eugenics, as practiced a few decades ago. The scientific consensus was that to prevent the spread of deleterious genes, drastic action, such as sterilization of the supposedly mentally defective, was justified. And there was compelling evidence for this – anyone who thinks that intelligence has no hereditary component, or that (in the West, at that time, and for that matter now) the fertility rate was lower for the intelligent than for the unintelligent is just kidding themselves. It follows inescapably, to anyone who doesn't deny modern genetics and evolution, that failing to act will result in lower average intelligence, with all the consequent bad effects on society.

    So do I favour sterilization of defectives? No. Leaving aside any moral issues with doing this even if the past scientific consensus were correct, the big question is: how large is the effect? Not that big, I think. Furthermore, the effect may well be ephemeral. Society is changing so fast that policies based on long-term effects of current social attitudes (that lead to differential fertility) are rather pointless.

    There's nothing new about these arguments against eugenics. All that's changed is that current political attitudes are no longer hospitable to the past "scientific" consensus, which has consequently changed.

  10. tgrass:

    Let's say you're an oncologist. There is a 74% consensus among your colleagues that people get cancer from the toxins contained in cigarettes. The consensus is significantly greater among specialists in respiratory cancers. It is also known that those toxins build up in peoples' bodies with each cigarette smoked, and take a very long time to dissipate. What would be your opinion on the subject? Would you advocate for alternatives to cigarettes?

  11. I think that there are two issues: first, giving a mean estimate of the thing you care about, say the global temperature in 50 years; second, giving a confidence interval for that estimate.

    I think that the "consensus" on any topic is probably a pretty good guide when it comes to estimating the mean. Surely it is right on average? But I think that the consensus is a bad guide when it comes to estimating the confidence interval because the scientists who form the consensus are horrible judges of their own accuracy (as are we all).

    My rule-of-thumb solution is to have my own 95% confidence interval include the mean estimates of all reasonable experts, defining reasonable as publishing in peer-reviewed academic literature. This makes me appear quite skeptical since I am always getting in fights with folks like Phil, not so much because I disagree with their mean estimate but because I think that their confidence intervals are way too narrow.

  12. From the 5 books I've read about string theory, I'm not convinced.

    But what I haven't seen in that field are scientists consistently adjusting the data up when it should obviously be adjusted down:

    http://www.ncdc.noaa.gov/img/climate/research/ush

    I haven't seen people say that some aspect of string theory will kill us all (with 90% confidence) unless we take trillions of dollars out of the world economy and give it to bureaucrats.

    I haven't seen proponents of alternative views of string theory have their careers sabotaged by a clique of 2nd rate scientists.

    And I'll freely admit that some of the more abstract math involved with string theory are beyond my training. That's just not the case with today's climate science where the "top scientists" either do not understand or choose to ignore observation, statistics, and the scientific method.

    Moreover the "creationist" lumping is really getting tired. If someone were skeptical of certain aspects of cosmology (like the existence of a massive amount of dark matter, for example), you wouldn't lump them in with creationists.

    Do you think Seth's (above) scientific acumen is on par with even the top creationists?

    Do creationist have talent on par with Richard Lindzen or Steve McIntyre pointing out how Evolutionary theory is wrong? Clearly not.

  13. David Kane: Your approach makes sense (at least as a general principle, without my having really thought about how it applies in this particular example).

    Z: I agree with you that string theory isn't a scientific consensus in the way that climate change or evolution are consensuses.

    I think the better analogy to string theory in physics is rational choice theory in political science: a research program that has huge gaps in its connection to reality while at the same time having a strong internal logic that makes it an appealing direction for research. There is far from a consensus that string theory (or, in political science, rational choice theory) is "correct" or even a fruitful research program, but the opposition to it arises, I think, not from claims of fraud and conspiracy but from a general competition for resources (faculty positions, research funding, etc.).

  14. But the consensus on climate change is not even close to the kind of consensus we have on evolution.

    That's propaganda.

    If the question is "Do you think Man's activities have changed the climate?"

    Then you'll have a consensus.

    If the question is: "Mean surface air temperature has increased by 0.3 to 0.6 C for the entire 20th century. The IPCC projects that warming to increase 10 fold to 0.3 C to 0.6 C per decade in the 21st century. Do you agree?"

    Then that "consensus" melts away . . . unless grant money is involved.

  15. Thanks for improving your comments about me. There is a lot of room for further improvement. If you've looked at my original post ("Three Things…") you don't show any sign of it. You say I should "talk to an actual expert". Having written a non-trivial paper on how modelers fool themselves. I am surely better equipped than you and Phil and quite a few physicists to see that the climate modelers are fooling themselves. To put it bluntly, in certain ways I am the expert that you and Phil should be consulting. It's fine that you don't — but then to criticize me for not consulting an expert is … absurd. If you take away the climate models it becomes really difficult to claim man has had a big effect on global temperature. Since the temperature now isn't strikingly high (compare to the Medieval Warm Period), and the rate of increase is no higher than it's been twice in the last few hundred years at times when man couldn't have been having a big effect, without the climate models it is hard to make any case at all that man is having a big effect.

    As for your example of how I pointed to a guy who studies sea levels, and trusted him over "expert consensus" you are failing to understand that the guy who actually studies sea levels is more of an expert on sea levels that those you call "experts".

  16. Z:

    My impression is that the consensus is pretty strong; you can see Phil's recent blog entries for more on that topic.

    Seth:

    Yes, I looked at your original post; that's why I wrote that you are interested in "more general issue of how scientists and others can be fooled into following a consensus." I agreed that you an expert in many things–but not in physics! When I see you writing about the Medieval Warm Period etc., what I see is you repeating talking points. Your link to "the guy who actually studied sea levels" came from a newspaper columnist. I'm pretty sure the people on the IPCC and other expert panels know about the medieval warm period and also have a lot of sea level measurements too. That one guy quoted in the newspaper column is not the only person who studies sea levels.

    I can believe that you've made important contributions to the study of depression and nutrition. As you've discussed in various places, your background as an experimental psychologist has helped you in gathering and interpreting your data and also in understanding the literature on the topic. But, no, I don't think you have anything special to offer when you start bringing up the Medieval Warm Period, sea level measurements, and the rest. Here I'll go with the experts.

  17. "Z" wrote: "If someone were skeptical of certain aspects of cosmology (like the existence of a massive amount of dark matter, for example), you wouldn't lump them in with creationists."

    Funny you should mention that.
    http://www.scientificamerican.com/article.cfm?id=
    discusses a gravitational theory from Petr Horava, a physicist at the University of California, Berkeley. If this theory proves to be more correct, then dark matter and dark energy MAY turn out not to be there.

    "Horava gravity may also create the “illusion of dark matter,” says cosmologist Shinji Mukohyama of Tokyo University. In the September Physical Review D, he explains that in certain circumstances Horava’s graviton fluctuates as it interacts with normal matter, making gravity pull a bit more strongly than expected in general relativity. The effect could make galaxies appear to contain more matter than can be seen. If that’s not enough, cosmologist Mu-In Park of Chonbuk National University in South Korea believes that Horava gravity may also be behind the accelerated expansion of the universe, currently attributed to a mysterious dark energy. One of the leading explanations for its origin is that empty space contains some intrinsic energy that pushes the universe outward. This intrinsic energy cannot be accounted for by general relativity but pops naturally out of the equations of Horava gravity, according to Park."

    I think it's fascinating that dark energy and dark matter went from being unknown to possibly most of the universe and now possibly to nonexistent.

    [Please note that I do realize I'm wandering completely off topic from climate science to cosmology, and I do know they aren't the same thing]

    [Horava's name has been converted to standard English characters.]

  18. Fantastic post.
    The limits of experts consensus, how to optimized expert consensus, is one of the most important topics in social epistemology in my opinion. I think this is well worth a paper or a book.

    I encourage you to repost this to overcomingbias and less wrong, too.

  19. To pick up on what Phil said:
    When the experts get it wrong, it makes a big impression. Why? Because usually, they are right. Against the invisible background of the expert consensus being right, say, 95% of the time, the few cases it turns out wrong are very noticeable. In this situation, many of us humans will find it impossible not to overestimate the frequency of the discordant event (expert consensus actually being wrong). From there, some take a fatal additional step, and conclude that any time there is a dissenting view, then surely experts must be wrong. Again.
    It's the wrong conclusion, but it is not really strange to see it be reached.

  20. Andrew,

    Since you write software, I think you should have a concern for whether the software used by climate "scientists" is accurate.

    I mentioned previously in response to Phil, and he didn't seem to care, that in order to get the hockey stick graph, Mann had to write his own FORTRAN code to perform principal components, and that his code produces a different answer than SAS, SPSS, S-Plus, R, etc.

    See page 17 of the below-linked document for a comparison of Michael Mann's data (which are of dubious value, but accepted for the sake of argument) run through his code and everybody else's code: the former gives a hockey stick, the latter does not.

    B. D. McCullough and Ross R. McKitrick
    "Check the Numbers: The Case for Due Diligence in Policy Formation" The Fraser Institute, February

    http://www.pages.drexel.edu/~bdm25/DueDiligence.p

    Regards,

    Bruce

  21. Jonathan writes:

    And I can't tell you how many people I've met who recite hideous science regarding vaccines and autism.

    There is no hideous science behind the view that vaccines may be linked to the rise of autism. In fact, your provocative words used to drown out any debate (i.e. "hideous") is the problem.

    Prof. Seth has been unfairly criticized and lambasted on his own blog when having doubts about the consensus regarding vaccines and autism, doubts for which he offers plenty of reasons. And no, the statistical and scientific data is NOT at all clear on this matter. There have been many studies, as well as that most scientific of things (personal anecdotes from highly observant parents) that would indicate the matter is hardly resolved. The scientific consensus wants to drown out any doubts that vaccines could cause or be linked to autism, yet does not have the evidence to do so. As of now, I have not seen ONE study looking at vaccinated versus non-vaccinated kids to see incidence of autism. Furthermore, the supposed cause, genetics, has been dismissed in EVERY meta-analysis of the disease as being anything more than coincident. In other words, it is clear that genetics has SOMETHING to do with autism, but it is also clear that genetics has SOMETHING to do with cancer or heart disease also. Clearly, in the latter, environmental factors may have a greater impact than genetics. That's also the case in autism. Looking at vaccines as the primary environmental culprit is a great idea, and all the scientific consensus can do is to denigrate it with extreme prejudice and a priori claims like it is "hideous".

    Shame on you, for abandoning the scientific method in favor of religious ideology pretending to be science.

  22. The real problem, Prof. Gelman, is that in many areas of complex science (e.g. climate studies, chronic disease, evolution, nutrition, economics, etc.), the real structural models posed by scientific reasoning is highly non-linear and often yields few precise predictions, but rather focuses on stylized "facts" about a phenomena. That is knowledge, too. It's just that it's not the same type of knowledge as Newtonian physics, where standard statistical models have a lot of power in determining the true of a posed model.

    Take nutrition or epidemiology, what goes for "scientific consensus" is routinely overturned, often dramatically. Eggs are good for you. No, eggs are bad for you. No, they're good. Wine is bad, wine is good. Carbs are bad, carbs are good. Animal (saturated) fat is bad, vegetable fat is good, even if it's hydrogenated. Animal fat is good, vegetable fat is bad. Eat margerine…NO, no, no, eat butter instead. Running prevents cancer… No, too much cardio and running CAUSES cancer. You get my point.

    So, then, the same people who are supposed arbiters of informed scientific opinion have the gall to claim that those who criticize the prognostications of such fields are "hideous" or full of it. It's sheer madness, and I would think that the Bayesian view would be best to avoid idiocy like this.

  23. Bruce: I agree with you that the details matter. I have not studied the details on this, and I'm trusting the judgment of Phil and others who have. But I certainly agree that it's good to have such scientific debates out in the open.

    Seth: I suggest you reflect upon your phrase, "the guy who actually studies sea levels," which at first seems like a bit of ironic wit until you realize that it carries with it the ridiculous implication that the generally-acknowledged experts in the field have not looked at sea level measurements.

  24. Andrew, this is from the article I linked to:

    When I spoke to Dr Mörner ["who for 35 years has been using every known scientific method to study sea levels all over the globe"] last week, he expressed his continuing dismay at how the IPCC has fed the scare on this crucial issue. When asked to act as an "expert reviewer" on the IPCC's last two reports, he was "astonished to find that not one of their 22 contributing authors on sea levels was a sea level specialist: not one".

    You are trusting the 22 contributing authors (none of whom is a sea-level specialist) over someone who is a sea-level specialist. That's what I meant. Yes, it is ironic.

  25. Andrew,

    Details do matter. And my point is that Phil has not studied the non-science details, e.g., the data and the code. He may be an expert in the theory of the problem, but as to the practical empirical aspects (accuracy of data and code) he knows next to nothing.

    All the theory in the world can be correct, but if the data and code are bad, then all is for naught. This is the point that I am trying to make and that Phil dismissed as an inconvenient truth (to turn a phrase).

    If you are relying on Phil's opinion, you should ask him how he knows that Michael Mann's data and code are correct. If he says that they are correct, refer him to the few pages in the document I mentioned previously.

    Bruce

  26. Seth:

    I think you're relying a bit too much on this newspaper column you linked to. The columnist appears to have strong opinions, which is fine, but I think it's a bit naive to take this guy's word for it that the IPCC doesn't know what it's doing. No, I don't think it necessarily makes sense to trust one particular "sea-level specialist" who happens to disagree with the consensus of other sea-level specialists.

    To get back to my original point here: the IPCC people and other expert commissions represent the true experts. Phil is enough of an expert to evaluate the competing claims. I'm not an expert. You're not an expert either, but you place a lot of weight on things you read in newspaper columns.

    I respect where you're coming from on this. Phil's a close friend of mine and I know him well enough to trust his judgment in this area. You don't know Phil so well, and you don't have a physics background, but you're personally aware of other settings where the scientific consensus seems to be wrong. Not knowing where to turn, you grab various blogs and newspaper columns that come your way to make a judgment. I don't think this is a good way to go–honestly, I think you'd be better off reading Wikipedia entries–but I respect that, without a "Phil" figure you can trust, you're left adrift.

    As discussed in my blog entry above, I don't have an easy answer for you on this issue of how to form opinions when you don't know any experts in the area.

  27. You disallow arguments by authority (I have a friend who is an expert and knows best), _force_ your friend to explicate the path from the data to the conclusions (maybe in an elegant way like Gardin's Logicist Analysis) and get other experts to stare at it.

    We are always going to be wrong, and can only get less and less wrong and the above might spead up that process.

    And the reality is that it needs to be delegated to sets people who will actively disgree and accelertae each others process of getting less wrong.

    So if you don't have resources its hard not to count on experts you happen to know.

    Keith

  28. Andrew:
    I'm not claiming to be an expert on climate change. I'm just telling people why I believe what I believe. Obviously I think I am thinking about this stuff the right way — if I knew a better way, I'd think that way instead — so if somebody wants to say "well, in absence of other information I'll take Phil's word for it" I guess I am OK with that. But it would be much better if everyone, including Andrew, would look for other sources of information too!

    Seth:
    Morner seems to be out there nearly on his own, in being a credible scientist who claims that the sea level is not rising. He is very much on his own in claiming that neither tide gauges nor satellite data can be trusted for measuring sea level rise. (I also cannot resist pointing out — I really did try to resist! — that Horner believes in dowsing, has written a book on the subject, and tried (and failed) to demonstrate the effectiveness of dowsing on a TV show; and…oh, never mind. I would say that this does call Morner's scientific judgment into question, but if he really knows his stuff on sea level rise then he should be taken seriously on that subject).

    Anyway…I don't know nuthin' 'bout sea level rise, other than what I read in the papers. Seth, I doubt you do either. One paper I just took a look at is Church et al., "Sea-level rise at tropical Pacific and Indian Ocean islands," Global and Planetary Change 53:155-168 (2006), which says, among other things, "Mörner et al. (2004) and Mörner (2004) recently drew attention to the potential vulnerability of the Maldives. However, Mörner et al. (2004) argued that there had been a 30 cm fall in sea-level at the Maldives over the last 50 yrs while Mörner (2004) argued that there had been no global averaged sea-level rise over the decade of the 1990s. Mörner's conclusions concerning a sea-level fall at the Maldives have been firmly rebutted by Woodworth (2005), Woodroffe (2005) and Kench et al. (2005). " Just for laffs, I took a look at the last of these…there's a lot of stuff about interpreting old coral formations, and where "beach rock" can develop, and so on…none of which I can follow at all. But my point is that it is not the case that sea level experts think the sea level isn't rising, and the IPCC is ignoring them. Morner is claiming the sea level isn't rising, and lots and lots of other sea level experts disagree with him; take his statements seriously and rebut them; and present their own data and analyses of sea level rise. If you want to claim that science is on your side because this sea level expert Morner says so, you should also explain why you discount the views of all of the other sea level experts.

    Bruce:
    Mann's original analysis method is known not to be correct, as I mentioned in a comment on the "Climategate" blog entry. In another comment on that entry, I said "I agree with Matt that "Everybody's decided to die fighting on Mount Hockey Stick, when the really difficult analytical and normative work is still ahead." Whether current northern hemisphere mean temperatures are the highest in 400 years, 1000 years, or 10,000 years is a pretty minor point, when what we really care about is the _next_ 400 years or 1000 years."

  29. Phil: Thanks for the clarification. When I described you as "enough of an expert to evaluate the competing claims," I was referring to your ability to navigate the climate science literature, which I think comes from your working in physics for many years. By analogy, I have enough social science experience that I can evaluate the social science literature to some extent–I can scan an article and get a quick sense of where it's coming from–even if it's not in an area that I'm an expert on.

  30. Prof. Gelman,
    By the way I'd like to see you give a series of posts where you attempt to grapple with how to optimize qualitative analysis.
    My sense is that among the spectrum of thinkers, you're a very clear-headed qualitative analyzer, and I like your technical critiques of display of statistical information.
    Given that I assume statistical papers have their fair share of prose, I'd like to see your analysis of that as well.
    And perhaps some recommendations on resources.
    I've done some googling, and it seems to me that qualitative analysis seems rather un-unified, unorganized, undeveloped, and maybe even under-researched, and to develop it as a formalized discipline would require drawing from a lot of fields that you have unusual literacy in.

    It also seems to me where a lot of myth-making, and non-transparent status seeking by subpopulations, attaches itself to purer empiricism in the social sciences.

  31. Climate scientist consensus is a double-edged sword.

    Suppose we take it as given that they agree on global warming, but that their models are too complicated for outsiders to verify.

    We then must base our opinion on their reputation.

    ClimateGate has shattered the reputation of a dozen or so of the top people in the field, and of the best of the three to five datasets on which everybody else bases conclusions. The direct effect is that we shouldn't trust what those dozen people say, or that dataset.

    The indirect effect is much bigger. What has the response of the other climate scientists been? Silence, or "that is normal science" or "it doesn't really matter because the conclusions are correct anyway." I find the silence or complicity damning evidence against that entire scientific community. Now, when they say something we can't understand, they should have *less* credibility than an amateur, not more. If they can support their conclusions with data and understandable theory, fine— otherwise, I'm not going to buy their used car.

  32. Eric R., you say "ClimateGate has shattered the reputation of a dozen or so of the top people in the field, and of the best of the three to five datasets on which everybody else bases conclusions." I don't think either of those is true, or even close to being true. Could you name, say, six people whose reputation has been "shattered"? Could you name two datasets that have been discredited?

    I haven't read the "Climategate" emails. I have read supposedly damning excerpts from them, at a "denialist" website…and there are two of those emails that I think indicate serious misconduct in refusing to share data. (Nothing that I saw suggests that any data were fabricated or that analyses were faked). Associated Press had some people read all million words of all of the emails; their conclusion (see
    http://www.google.com/hostednews/ap/article/ALeqM… ) is "E-mails stolen from climate scientists show they stonewalled skeptics and discussed hiding data — but the messages don't support claims that the science of global warming was faked, according to an exhaustive review by The Associated Press."

    So, Eric, perhaps the reason condemnation of the Climategate scientists hasn't been more widespread in the scientific community is that the scientists actually didn't do much that was bad. I think you're just wrong about what the emails show. Enlighten me.

Comments are closed.