Statistics is hard etc, again

Michael Nelson writes:

In a comment on your Saturday post, you linked to a previous post on Brandolini’s law, which I followed and read. I was especially struck by the post script, where you quote a commenter who gave a clever counterargument: it’s easy for mathematicians to disprove BS proofs, much easier than it is to write a proof that looks superficially correct. The commenter pointed out, too, that most statistical errors are relatively easy to spot and counter. It made me wonder why we (statisticians and methodologists) have this problem of disproving BS and they (mathematicians) don’t.

The answer I came up with is disconcerting. It seems to me that mathematicians are the principal gatekeepers for mathematics, even among researchers in other fields that use mathematics; while statisticians and methodologists are not the gatekeepers for statistics and methods among social scientists who use them. In other words, it takes less work for a mathematician to unmask BS because non-mathematicians who use math understand the distinction between an expert in mathematics and an expert in physics who uses mathematics. They get that math is hard, that it requires special knowledge and study, which most people in other fields don’t have. Many social scientists have no such understanding of statistics. You often say that statistics is hard. I suspect many social scientists would read that statement and think, “Yep, it takes real expertise like I got in grad school. Sure, I only took two or three stats courses, but my advisor and older labmates were able to show me exactly how to do statistics in our field.”

But that’s not the disconcerting part to me! I fear that they don’t take us seriously as experts because we literally teach them not to. I think there’s a strong argument to be made that we (statisticians) routinely teach statistics to future social scientists as a tidy package of simple techniques, and we strongly imply that that’s all they need to know. We do this because, I believe, we know that social scientists generally aren’t expected by their programs to have quantitative rigor. We know that statistical expertise takes time and practice, but we also know that other programs are only going to make so much room for coursework that isn’t substantive. Yet they still expect us to certify their students as qualified in very technical methods. We know the professors in other departments expect us to include the methods they use (NHST) with only the briefest of caveates. We know there are labs all across campus with psychologists and sociologists and so forth who practice statistics like it were witchcraft, with mysterious formulas superstitiously adhered to, or worse, like it’s something the software does. And, for the most part, we pass the students, we shrug at our colleagues’ ignorance and propagation of ignorance, and they all see us doing it. They learn what we teach them, by curriculum and by example. We never teach them that they aren’t experts. Then we get mad when they ignore our expertise!

Imagine if every stats syllabus listed among the course goals that passing students will “understand the limits of their own statistical knowledge,” will be able to “recognize circumstances in the design and analysis of their studies when consultation with a statistical expert is necessary,” and will “respect the expertise of the statistician to the same extent that they expect the statistician to respect the researcher’s substantive expertise.” Only in that world is it truly reasonable to expect social scientists to listen to our critiques, much less to do statistician-quality research.

My answer is . . . I’m not sure! I feel queasy about proposing that anyone be the gatekeeper of anything.

I’m posting here in case any of you have thoughts on the matter.

One other thing. Nelson writes, “The commenter pointed out, too, that most statistical errors are relatively easy to spot and counter. It made me wonder why we (statisticians and methodologists) have this problem of disproving BS and they (mathematicians) don’t.” But . . . lots of well-credentialed statistics experts make basic statistical errors. The foundations of statistics are a mess, and over and over again I’ve seen big-shot statistics professors making what I consider fundamental misunderstandings. Statistics errors may be as easy to spot as math errors, but they’re also much easier to make. I guess that statisticians are typically better at statistics than non-statisticians—for one thing, we typically have less invested in our substantive conclusions, which I think would make it less likely for us to use sloppy reasoning to support our arguments—but, still, the whole field is just less codified. There are lots of places to get confused. So I don’t see any easy answers here.

41 thoughts on “Statistics is hard etc, again

  1. Almost right away in stats 101 we start out testing a model that predicts zero difference between group A and B.

    This is contrary to science but it is also difficult for the student to understand why everyone is doing it. The confusions then stem from there.

  2. The very first sentence, that of Michael Nelson,

    “In a comment on your Saturday post, you linked to a previous post” has me in a time warp because

    1.the first reference is to October 30, 2021

    and

    2. the second reference is to January 28, 2019.

    I suppose such problems are a result of the time lapse between blog scheduling and actual postings. Covid and the odd invasion tend to get in the way.

  3. Disclosure: I do not have any statistics degree (PhD in economics) but have taught statistics at both the undergraduate and graduate levels for many years. I don’t pretend to know more than I do- I am confident in my ability to work with data, but when it comes to understanding the technical issues associated with different error distributions I know I lack a lot of background knowledge. So, if the thrust of this post is that I should not be teaching statistics, then I plead somewhat guilty.

    “Somewhat” because: (1) as Andrew says, those with statistics degrees make plenty of mistakes (some of which I would not make), and (2) because I don’t think restricting access to statistics is the right thing to do. I actually want to democratize statistics – I’d like to see more people doing statistical work, but without requiring them to have a graduate degree in statistics. I guess the question is whether that is possible – can we tolerate having people use statistics that do not have extensive training in statistics?

    I want to say “yes,” although I am open to hearing opposing arguments. Data permeates virtually all disciplines and I can’t imagine restricting statistical work to only those with extensive training. Nor can I imagine requiring that a “real” statistician be required for any project involving data (great for the discipline, and even a good thing, but quite unrealistic). That leaves 2 realistic options: much less research involving data, or have people work with data that do not have enough background to do so.

    I advocate the second, but with two caveats. First, we can improve statistical education (as any reader of this blog knows, there is widespread agreement on that). And we should improve it. Second, much of the training that statisticians receive is simply unnecessary to doing good statistical work. So, I had to prove the Central Limit Theorem in 3 undergraduate and 2 graduate courses that I took. Those proofs did nothing to improve my knowledge of how to use it or when not to trust it (e.g., I had to learn through a consulting project that the usual sample sizes required to rely on the CLT (please excuse the word “rely” as I know not to rely on anything as the word is usually used) do not apply in the presence of extreme kurtosis).

    Statisticians, as with most disciplines, have extensive training in what their discipline has decided is required for an academic – much of this has to do with protecting the field against too much entry (by “unqualified” people). Much of that training is simply unnecessary (I can speak to this for economics and I believe it to be true for most disciplines, including statistics). I wish I had more statistics background, but I’m not convinced that the typical graduate courses would provide what I would find particularly useful – hence I educate myself through other channels.

    Sometimes, it seems to me that statisticians are saying that people shouldn’t practice statistics unless they are trained statisticians. But statistics is too important to be left to statisticians! I’d say the same thing about economics.

    • Dale – I have been struggling to develop material to help folks grasp the logic of statistics without needing the usual graduate training that does not provide that but only facilitates it with a lot of experience trying to apply it.

      Send an email if you wish to discuss – but a teaser is below:

      The way in which I try to cast statistics (as of a couple months ago):

      Science is thinking, observing and then making sense of thinking and observing for some purpose. Making sense is formalized in assessments of what would repeatedly happen in some reality or possible world. If it would repeatedly happen (a habit either of an organism, community or physical object/process), it’s real.

      Statistics can be defined as formalizing ways to learn from observations using math (i.e. using models). Unfortunately what is taken as abstract statistical work is too often misunderstood mathematics where it’s representational role has been suppressed or overlooked. Models or assumptions are often just taken as the price one has to pay to obtain statistical outputs like p-values, confidence intervals and predictions. Unavoidable just like taxes. However, they are representations of some reality beyond our direct access. Even if only implicitly.

      It is always the case that given we have no direct access to reality, reality must be represented abstractly in our heads. Given that we must think about reality using abstractions, we can only notice aspects of those abstractions. These need to not be confused with the reality they just attempt to represent, in some meaningful way for some purpose. The veracity of any statistical method depends on an implicit abstract fake world being not too wrong. That is too different from the “real” world for the given purpose in some meaningful way.

      Mathematics is required to discern exactly what an abstract object or construction implies. Taken as being completely true, what follows? However, mathematics as it is usually taught and written about can be a formidable barrier for most. Even those with an undergraduate degree in mathematics may have only learned to “get used to it” rather than actually understand it. That is, they can do the calculations correctly but do not know what to fully make of the results. But mathematics has many mediums and one in particular can perhaps be grasped most widely by researchers. This is because, as CS Peirce pointed out, the object of mathematics is some or all hypotheses concerning the forms of relations in the abstract construction. All mathematical knowledge thus has a hypothetical structure: if such and such entities and structures are supposed to exist, then this and that follows. Fortunately there a many ways to discern what follows.

      CS Peirce further defined mathematics as the manipulation of diagrams or symbols taken beyond doubt to be true – experiments performed on abstract objects rather than chemicals – diagrammatical reasoning. Here diagrams more than symbols have been argued to be more perspicuous (an account or representation more clearly expressed and easily understood or lucid). Diagrams are arguably the medium of mathematics most can grasp. An abstract diagram is made, manipulated and observed to understand the diagram much more thoroughly. Recently they have been accepted in the mathematical community as being rigorous – Visual Reasoning with Diagrams https://link.springer.com/book/10.1007/978-3-0348-0600-8

      By far the main mathematical constructions in statistics are probability models. These can easily recast as diagrams which then can easily be automated and animated with computer simulation, given modern computation. Once probability models are recast as diagrams then then the diagrams themselves can be used to generate the pseudo-random variables needed for simulation. This generating can be inefficient but is valid and much easier to grasp than other ways. This transforms the understanding of probability into experiments performed on diagrams just using simulation all the way down. Those abstract mathematical probability models are best understood in terms of what would be repeatedly drawn from them and simulation does the repeated drawing. I believe that simulation provides a profitable mechanical way of noticing aspects of probability and statistics where the learning about a model is clearly and fully distinguished from what to make of observations in hand. And it involves very little mathematical skill but rather just the ability to think abstractly. But there is no free lunch, it needs to be worked with, experienced and reflected on.

      Models take elements and relations among them in the represented world [that produced the data] and map them onto elements and relations in the representing world [probability world]. This requires transporting what repeatedly happens given a model, to what reality happened to produce this time. In the second we can know exactly what we are learning about (the probability model), in the first (the world) we can only guess or profitably bet about it. Those guesses are informed by what repeatedly happens in the probability world. However, it is really just replacing the medium or form of mathematics (a means to understand an abstract representation, the aim of which is to infer necessary conclusions from hypothetical objects) with something more concretely experimental and hence potentially self correcting with persistence. Thereby better facilitating self verification more widely with less mathematical skill.

      • > CS Peirce further defined mathematics as the manipulation of diagrams or
        > symbols taken beyond doubt to be true

        Mathematics is using logical reasoning to show that statements of the form “If A, then B” are correct.

        > Recently they have been accepted in the mathematical community as being
        > rigorous

        I haven’t read the book, but a picture is not a proof. A picture may suggest the reasoning to you.

        • The book and other papers challenge statements like “but a picture is not a proof. A picture may suggest the reasoning to you” that was the prevailing view in the past. On the other hand there might be advantages to staying more vague than a written proof.

          > Mathematics is using logical reasoning to show that statements of the form “If A, then B” are correct.
          Yes, a convenient diagram there is a Venn diagram.

    • I feel the opposite where I have trouble convincing both private and public organizations that I have value to add as a statistician with a master’s degree beyond the statistical knowledge other analysts and researchers possess. This specialized knowledge is by no means all-encompassing but it does provide extra insight in many situations.

      That being said, I know there are plenty of non-statisticians (including but not limited to psychologists, economists), particularly at the Ph.D. level, who are legitimately statistical experts. However, having a team member with statistical expertise beyond the normal prerequisites (though not necessarily a full degree) can be extremely useful in both private and public research. This additional benefit is not always recognized.

      I have a lot more thoughts about this topic, including a defense of some degree of gatekeeping, but too for one reply.

  4. > most statistical errors are relatively easy to spot and counter

    I think the Hot Hand story is a good counterexample. I think if I didn’t know the answer the original method sounds hand-wavy reasonable enough that it (or a variation) would slip by me (not that this is a terribly high bar).

    > I feel queasy about proposing that anyone be the gatekeeper of anything.

    Agreed.

  5. Michael Nelson wrote, “It seems to me that mathematicians are the principal gatekeepers for mathematics”.

    I don’t understand this statement. Anyone can check a proof. You don’t have to get a degree in math to give you a license to check a proof. Admittedly, journal articles are usually written assuming the reader already knows certain things. But, that’s why people write books: To organize a subject and make it easier to learn.

    Andrew wrote, “The foundations of statistics are a mess”. If you teach your subject in an illogical way, then why do you expect people to recognize errors? Math textbooks all agree on what is correct. That’s because math depends on logical reasoning. Most statistics textbooks are full of illogic. If you build your field on illogic, then don’t be surprised when users simply plug their numbers into your formulas and believe what comes out.

    Michael Nelson wrote, “it’s easy for mathematicians to disprove BS proofs”.

    If I give an example to show a math argument is fallacious, that ends the discussion. If I give an example to show that unbiased estimators can give ridiculous results, some people say the example is bad or irrelevant.

    • ” If I give an example to show that unbiased estimators can give ridiculous results, some people say the example is bad or irrelevant.” I agree with this, but I think it’s even worse than that. If a statistician says an estimate is unbiased, then people think “then that’s what I want to use.” Pointing out that unbiasedness might be the 10th most important quality of an estimate to their specific problem meets with puzzlement. Statistics, unlike mathematics, has an implied metric of practicality for particular problems that has nothing but handwaving underlying it. There are no possible theorems for when to use a linear regression to get an estimate.

    • I agree, I can’t actually think of many examples where Mathematicians do act as principal gatekeepers for mathematics. Economic Theorists write proofs (as do Theoretical Econometricians) and I’m not aware of any Economics department that regularly consults the Math department on research. Yet there doesn’t seem to be a large number of faulty proofs accepted in Economics (faulty as in the proof itself is invalid).

      Given how often researchers refuse to accept/admit their mistakes it’s worth considering that the problem stems more from something like a need to publish positive results rather than an inadequate understanding of statistics. Perhaps we can identify a field where applied statistics are employed well, and see what the differences are there.

    • “If I give an example to show a math argument is fallacious, that ends the discussion. If I give an example to show that unbiased estimators can give ridiculous results, some people say the example is bad or irrelevant.”

      But no statistician would claim that an unbiased estimator is *always* good. And no alternative method is always good either. Counterexamples work differently in statistics (as far as they’re not purely mathematical). We have to use methods that in certain situations will not work well, and we don’t always know that we’re in such a situation. Counterexamples are fine and may be relevant, but they can’t kill a statistical approach as they could kill a theorem.

      • By the way this is a major issue where proper statistical expertise enters: Awareness for under what circumstances an approach will be problematic, what to do about it, and potentially also awareness that we have to live with problematic approaches due to for example non-identifiability of certain problems with model assumptions.

      • Often the examples show that the basic reasoning is faulty. But, many statisticians say that the method is really a good method, just not good in the particular example. I.e., they say what you just said. If your method doesn’t work in simple examples, why do you think it works in complicated ones?

        • David:

          I think there are some bad statistical methods out there, or methods that do more harm than good, or whatever. But “unbiased estimation” is not a statistical method, it’s a principle for evaluating statistical methods. As a principle, it makes sense in some settings and not others. It’s not a universal principle and, as Christian says, everybody knows that, but there is a problem that it’s misunderstood and thought of as more universal than it is. To answer the question posed in your last sentence, the unbiased estimation principle does work in some important simple examples.

        • If the principle makes sense only sometimes, then it isn’t really a principle. Mathematicians have heuristics too, but then they use reasoning to make sure that they haven’t made a mistake.

          The posterior mean is unbiased in parameter space. That’s a principle that works. It is unfortunate that the word “unbiased” was used for something else, thus promoting an error to a principle.

          Imagine a world where computers that could handle Bayesian calculations existed in 1900. Would we be having this discussion? Why keep teaching all these “principles” that are so often “misunderstood”?

          Mathematicians are also better at stating the hypotheses that they need for a result, but I don’t think that is the main problem here.

        • David:

          OK, let me not say that unbiased estimation “makes sense in some settings and not others,” as this seems to be causing confusion. I was using “makes sense” with the meaning of “sensible,” not “makes sense” with the meaning of “have a clear definition.”

          To stick with your terminology, I’ll say that the principle of unbiased estimation always “makes sense” in that it is clearly mathematically defined, but that using this principle to decide what statistical method to use is not always a good idea. In some settings, including some important simple examples, it is a good idea; in other settings, not so much.

          In statistics I’d say it’s heuristics all the way down.

        • > “unbiased estimation” is not a statistical method

          Wouldn’t you say that “minimum-variance unbiased estimation” or “best linear unbiased estimation” are statistical methods? If MVUE and BLUE (and MLE?) are not statistical methods, what do you consider statistical methods?

  6. I think a big part of the problem is that statistics, as an applied field, is open in a way that some others aren’t. By that I mean that, while some models and interpretations of models can be ruled out purely on the basis of logic, a plethora of potentially valid options is usually available. For this reason, good statistical work, as I understand it, is usually allied to substantive knowledge about the question being investigated, including the population we want to generalize to, the data collection methods, possible factors that might affect outcomes, the prior experience of other investigators, etc. An ideal statistics team would have expertise both on the pure methods side and the substantive knowledge side.

    But a lot of teams are not ideal. Perhaps thinking they can get away without deep statistical expertise on the problem they’re working on, a lot of researchers in the social and natural sciences (and related applications) cut that corner. Many aren’t successful, and of course they don’t have the background to know how unsuccessful they are.

    OTOH, some statistical issues germane to a particular substantive field may have been worked over so many times that a careful practitioner with subject knowledge can cover the stats aspects as well. Just not always….

    • Peter Dorman said,
      “I think a big part of the problem is that statistics, as an applied field, is open in a way that some others aren’t. By that I mean that, while some models and interpretations of models can be ruled out purely on the basis of logic, a plethora of potentially valid options is usually available. For this reason, good statistical work, as I understand it, is usually allied to substantive knowledge about the question being investigated, including the population we want to generalize to, the data collection methods, possible factors that might affect outcomes, the prior experience of other investigators, etc. An ideal statistics team would have expertise both on the pure methods side and the substantive knowledge side.

      But a lot of teams are not ideal. Perhaps thinking they can get away without deep statistical expertise on the problem they’re working on, a lot of researchers in the social and natural sciences (and related applications) cut that corner. Many aren’t successful, and of course they don’t have the background to know how unsuccessful they are.

      OTOH, some statistical issues germane to a particular substantive field may have been worked over so many times that a careful practitioner with subject knowledge can cover the stats aspects as well. Just not always….”

      This does touch on a lot of good points. My own attempt at addressing these problems has been to design and teach a “continuing education” course, which has turned out to attract a lot of enrollees. The course notes for the last time I taught it are at https://web.ma.utexas.edu/users/mks/CommonMistakes2016/commonmistakeshome2016.html . (The course has been passed on to others since, and may have been modified). The title of the course seems to attract people’s attention, since most people don’t like making mistakes if they can’t help it. One of the first times I taught it, one of the “students” was the head of the state Department of Health and Human Resources; she brought along a colleague, and I believe that every time I taught the course thereafter, she sent two of her staff members. One of the things I point out early in the course is that statistics inherently involved uncertainty; we can’t expect to remove it, so need to figure out how to take it into account.

  7. I’d be curious to know the definition of “statistics” that different commenters are working from. How does it differ from “data analysis” or “modeling” or “machine learning”? To me the critical distinction is the focus on quantifying uncertainty (I’d be interested to know if others agree).

    Some of the friction with other fields comes from the simple fact that a lot of people really don’t care very much about quantifying uncertainty (or maybe ball-park uncertainty is good enough). Couple that with the fact that credible error bounds are almost always a degree harder to generate than point estimates, and there’s a lot of room for confusion (and feeling under-valued and so on) when working across fields.

    A lot of the difficulties relative to math might just come down to being application-oriented. In applications, you don’t really have the option of saying, “well, that case doesn’t fit the hypotheses so the theorem doesn’t speak to that” and walking away. You have to come up with your best answer (and hopefully some assessment of uncertainty), and this can involve trade-offs and judgment calls.

    And then there’s the problem of clinical trials! The kind of trade-offs-and-judgment sloppiness I suggested in the previous paragraph would (supposedly) never fly in that context. But the tidy thinking of clinical trials pervades a lot of statistical teaching, and all manner of problems arise when tools from that setting are deployed without guardrails.

    • Josh said, ” I’d be curious to know the definition of “statistics” that different commenters are working from. How does it differ from “data analysis” or “modeling” or “machine learning”? To me the critical distinction is the focus on quantifying uncertainty (I’d be interested to know if others agree). ”

      I agree!

      (Related: I often have told my students that “The biggest mistake in using statistics is expecting too much certainty.”)

  8. I am doing honours in sociology and there was no statistics module with it. However, in psychogy there was a statistics module and did very well in it. Statistics should be more intergrated into sociogical studies…

  9. I don’t think statistics is that hard. There are thousands – tens of thousands even? – of people with PhDs in statistics. It’s no more or less doable than any other skill or profession.

    Among the papers that are frequently referenced here as problematic, very few of them (any?), have *only* statistical errors, and almost none of them could be fixed by better analysis of the same data. The most egregious problems with almost all of these papers are in the assumptions implicit in the experiments, rendering the data useless to any kind of analysis, regardless of its quality.

    I’m not sure that these problems can be solved by better teaching of statistics. What people need is a better understanding of science. What’s students need is more challenging and scientifically-focused coursework across the board, taking into account the fundamental problems confronting social science experiments. Until this is solved, better teaching of statistics would do anything to improve reliability.

    • Jim:

      I disagree with your statement that statistics is “no more or less doable than any other skill or profession.” I think statistics is easier than physics and harder than . . . well, let me not cast aspersions on any other professions, but just say that it’s not unreasonable to suppose that professions vary in difficulty.

      Also in response to your second paragraph: yes, the problems we discuss typically are not just statistical problems, but statistical confusion is part of it. Just for example see the discussions here and here. In both cases, the work we criticize has conceptual/scientific/theoretical problems. But there are also statistical misunderstandings that are leading researchers, sometimes very accomplished researchers, astray. So I do think there’s a problem that people underestimate the difficulty of statistics, leading them to think they understand what they’re doing, when they don’t.

    • Jim said.”I don’t think statistics is that hard. There are thousands – tens of thousands even? – of people with PhDs in statistics. It’s no more or less doable than any other skill or profession. ”

      Quantity is not the same as quality — and the problem with statistics is that it is often taught in a watered down way — even at the graduate school level. One particular deficiency in teaching statistics is that the focus may be on doing computations, but that is the least difficult part of statistics. (As I have often told my students, “If all you can do is computations, you can be replaced by a computer.”) The most important part of doing statistics is finding a method whose model assumptions fit the problem being studied. This can not be taught in a routine way; students need to think carefully about the specific context — what can be assumed, what cannot be assumed, and what methods fit these restrictions.

      • “the focus may be on doing computations”

        That’s so funny! My first stats class was in a grad class in my discipline. I got 60/100 on the first assignment because I mistyped some of the data into the spreadsheet. I used the formula correctly. Hilarious. Needless to say I had no further respect for that professor. He wrote an intro level text book, which I later had the chance to get rid of for a class I was teaching. Small paybacks.

        “The most important part of doing statistics is finding a method whose model assumptions fit the problem being studied.”

        If you ask me, that’s science. You have to understand the reality and deploy the model to mimic the reality as closely as possible or dice up the reality into parts you can handle.

        • Jim said,
          “If you ask me, that’s science. You have to understand the reality and deploy the model to mimic the reality as closely as possible or dice up the reality into parts you can handle.”

          That’s applying statistics to science. You have to understand the possible models, and why they are suitable in some circumstances but not in others. For example, the Central Limit Theorem can be used to show that a Normal distribution is appropriate for certain circumstances; with a little more math, it can tell you that lognormal distributions are appropriate for other circumstances. (See, e.g., the handouts “Lognormal Distiributions 1” and “Lognormal 2” at https://web.ma.utexas.edu/users/mks/ProbStatGradTeach)/ProbStatGradTeachHome.html ).

      • +1 to Martha’s comment. Ideally, computation is in service of insight, rather than rote recipe application. NHST has been so super destructive to grad stats education it’s almost impossible to exaggerate.

    • Jim said, “Among the papers that are frequently referenced here as problematic, very few of them (any?), have *only* statistical errors, and almost none of them could be fixed by better analysis of the same data. The most egregious problems with almost all of these papers are in the assumptions implicit in the experiments, rendering the data useless to any kind of analysis, regardless of its quality.”

      I sometimes think the truly knowledgeable statisticians, the ones who can be counted on NOT to fall into the conventional traps of rote statistical practice (NHST, forking paths and so forth), tend to fall into the opposite trap. If you’re equipped with a huge range of sophisticated statistical methodology the trap lies in thinking all that machinery can turn crap data from a convenience sample into scientific insight.

      Statistics is definitely hard. But designing experiments in accordance with actual scientific rigor and gathering valid data that can be generalized beyond the sample at hand is massively harder than doing the stats afterward.

  10. I think there is some confusion between the need for “deep statistical knowledge” and the need for statisticians. I accept the former – and we all need more statistical knowledge (I’m not as sure about the “deep” part since it depends on what that means), but whether that need would be well served by having more statisticians is not as clear to me. More “good” statisticians, certainly. But more people with a statistics degree?

    I also question the distinction between content area knowledge and statistical knowledge. Specialization makes sense to a degree, but I think we have gone well past the point of diminishing returns. For many problems, the types of issues that jim is alluding to are not strictly speaking “statistical” but nor are they separate from statistics. If the concern with measurement is real, then content knowledge is required, as is statistical expertise. Having well functioning and well staffed multidisciplinary teams may be the ideal, but in the less than ideal real world I’d rather see more and better training in statistics for non-statisticians than more statisticians required on research teams (although I suspect with better training of non-statisticians, increased use of statisticians would be one result).

    • This circles back to Martha’s excellently simple statement just above: “The most important part of doing statistics is finding a method whose model assumptions fit the problem being studied”. Finding a method and understanding the model assumptions: that’s statistical knowledge. Knowing whether those assumptions are consistent with the problem being studied: that’s content expertise. Knowing whether your statistical method “fits”, and what that means: that’s statistical expertise. It depends on the problem: sometimes statistical knowledge and content expertise are enough; sometimes statistical expertise is necessary.

      Design your research teams accordingly. Content experts do need enough statistical knowledge to be able to tell whether they need statistical expertise.

Leave a Reply

Your email address will not be published. Required fields are marked *