Roger Henke writes:
I have somewhat of a background in broad strokes policy research. My knowledge of research methodology and stats is very limited and in hindsight I am quite flabbergasted by some of what I’ve claimed in the past based on questionable to say the least data and approaches and equally so by the lack of scrutiny those data and approaches were subjected to by peers, journals, and funders/commissioners of the ‘research’. The blog is a great reminder of those limitations, and of all that is wrong with the way a lot of social science is done. Quite a lot of the posts are beyond my technical understanding but as teachers of humility, open-mindedness and dealing with information labelled social science I value the blog greatly,
Another blog that I subscribe to just published this piece.
Given the intended audience of this blog, this kind of advice regarding statistics feels so utterly wrong that I am starting to wonder if all the fuss about change-finally-has=come of the last decade is an info bubble impression, totally disconnected from reality. Am I reading this advice the wrong way?
I took a look at post in question, by Danielle Bodicoat, which is called “How to get confident with statistics” and gives the following suggestions:
You are more than capable of using statistics
1. Know that we all Google stuff
2. Read the statistics analysis sections of papers
3. Use statistics as often as you can
4. Practice talking about statistics
5. Ask questions
I then replied to Henke, asking why he thought the advice there is so wrong? It seems ok to me. I agree that statistics is hard, and she could emphasize that—it could be misleading to say “It’s that simple,” etc.—but the specific advice to look things up, practice a lot, etc., seems reasonable.
Henke responded:
Yes, you are right, about her specific advice, but it is all based on the assumption that some stats courses during one’s coursework for becoming a research in social science subject X are sufficient to get a good grasp of the basics, good enough to be able to really understand if all kind of wizardry that programs like SPSS allow you to do is allowed, in the sense of your data etc etc conform to the assumptions underlying the wizardry, in your specific case, and to be able to self teach by way of sources one googles, etc. The Feyman’s of this world may be able to grab a book and master the subject on their own, but I am quite pessimistic about how many of them walk the planet. My reading of the widespread misuse of stats in social science is that it is mostly caused by people – like me – who just don’t know enough to responsibly apply the methodologies and analytic techniques that they are able to mechanically execute. I have come to think of research methodology, including everything to do with being able to properly assess data quality etc. plus analytic techniques to draw inferences from one’s material and the math underlying them as something that requires an educational preparation at the level of a specialized MA. Someone not able to really understand your textbook e.g. at the level of being able to discuss the assumptional rationales and possible alternatives of specific details of it with a fellow researcher/statistician runs considerable risks of misusing the toolbox. An obvious solution is requiring any project to have a stats advisor on board. That in itself would prevent lots of crap being produced. But in my pessimistic opinion one only gets close to an optimal situation when both conversation partners know enough about each others field of expertise to have a real conversation, implying that the methodology/stats advisor knows enough of the field the data come from to really understand the kinds of distributions one may or may not assume, and e.g. knows enough to engage in a discussion about how sensible subject-matter theoretical assumptions are underlying the project, and the subject matter expert knows enough to ask the right questions and is able to understand the suggested techniques in enough depth, when they are being explained, to know what can and cannot be said after applying them.
So yes, her advice to research students makes sense from one perspective: they really need to know a lot more about stats and that requires a lot of study, but I am pretty sure that anyone ‘mastering’ some basics and then going to the SPSS website to find out how to run a regression doesn’t really know what they are doing.
And the one piece of advice that I actually disagree with, given the current state of affairs in the social sciences, is that students should read stats analysis sections of papers to learn what others do (she calls it patterns), and how to write a good one yourself. Most others should start doing something different isn’t it?
To which my reply is: Sure, if you take Bodicoat’s post as saying, “Do these 5 steps and you’ll be a stat wiz,” then, yeah, that would be bad advice. But I don’t see that as her message. Rather, I see the message as, “Like it or not, you’re gonna be doing a lot of statistics, so take it seriously, and start now by reading up and getting lots of practice. Yes, you are more than capable of using statistics, but that doesn’t mean that statistics is easy. Statistics is hard, it’s not cookbook-codified, so if you want to use statistics—and you’re gonna have to—now’s the time to get going and learn.”
There’s always this balance in learning a skill, where you need enough confidence to try it, while still knowing your limitations. I think the steps of reading, practicing, talking, and questioning are a good way forward. The challenge of giving this advice is that you want to empower people without getting them overconfident, which is Henke’s concern.
I think the biggest problem is still that we teach frequentist NHST stats as the default in most courses. Whereas what we really want is to teach mathematical model building. 80% of what people need to know is algebra, calculus, and logic, 20% is a bestiary of common distributions together with Bayesian computation in something like Stan, and 0% is t tests, wilcoxons, Anderson-Darling, F tests, how to read ANOVA tables, partitioning sums of squared errors, permutation tests etc. But the 0% is 100% of what’s taught.
If there were one suggestion I would make it’s never take a stats class, but take a well taught course in ODEs, agent based models, and dynamical systems, that uses Bayesian methods along the way.
+1
Do you have any syllabi or book/article suggestions for such a course? I have some knowledge of dynamic systems and a little knowledge on practical implementations of Bayesian methods, but I know basically nothing about ODEs and agent-based models.
I’d strongly recommend both Uri Wilensky and William Rand’s “An introduction to Agent-Based Modeling” and Steven Railsback and Volker Grimm’s “Agent-Based and Individual-Based Modeling” for computational agent/based models.
I also thing everyone should read Henle and Kleinberg’s “Infinitesimal Calculus” from which you can both learn calculus and learn how to think using infinitesimal numbers and why the usual mathematician’s view of analysis (with measure theory and such) is superfluous for the applied practitioner. Newton had it right the first time, it just took 300 years for someone to invent the required formal logic.
For ODEs, once you understand Infinitesimal Calculus, it’s just basically algebra, but in ODEs most of the books are either pure math from a non-infinitesimal approach, or are physics books so I don’t have a good suggestion there per-se. But I can suggest both Fowler’s “Mathematical Models in the Applied Sciences” and Howison’s “Practical Applied Mathematics” as general purpose introductions to mathematical modeling, and they’ll talk a bit about ODEs but you’d probably need to know something about them to get the most out of it.
Also, everyone should be absolutely required to read and practically memorize by heart GI. Barenblatt’s “Scaling” (which is quite short for something so incredibly important)
It may also be useful to read H Jerome Keisler’s “Elementary Calculus: An Infinitesimal Approach” https://people.math.wisc.edu/~keisler/calc.html
Sorry, I should have said Leibniz was right. Newton had a kind of limit based thinking, Leibniz was the one who worked with infinitesimal numbers. But also, so did Archimedes almost 2000 years earlier. Henle and Kleinberg introduce these ideas and a little history in their first chapter.
Wonderful! Thank you very much, Daniel.
Wow, I haven’t read a single one of these books. Thanks for the references; they look really interesting and useful.
Daniel, you will like this work by one of my postdocs: https://psyarxiv.com/dtazq.
This is exactly the kind of approach that you believe is important to teach (to my utter amazement, this paper was rejected after review from the ironically named MIT Press journal Open Mind).
But really, for anyone to learn to work with modeling, significant preparation is needed in math. The idea that one can outsource the details to an expert, as someone suggested in the comments, is a pipe dream.
Shravan. I love that paper. I imagine that paper is probably one of the most important papers written in your field in quite a while. It takes the basic idea that there are ambiguous parses, but then once something is disambiguated, it’s committed to and can’t be backtracked (ie. in Prolog, a “cut” would have been applied) … and puts a mathematical model for the timing of that process on top. If that hasn’t been done already, then it seems like it must be a very big step forward. Congrats. I’m not surprised, but still disappointed that “Open Mind” would reject it. What was their logic?
Also Shravan, if you read the Henle and Kleinberg book please write to me and let me know what you think.
“I think the biggest problem is still that we teach”
Too much statistics and not enough science.
Fascinating topic and discussion, by Daniel and others below. As someone who does a lot of planned experiments (well, OK, my students and postdocs do them), I think that I agree with the spirit of Daniel’s comments but in practice I would never follow that advice. Even though the frequentist methodology is truly stunning in its madness and stupidity, not to mention irrelevance for the question at hand, it is a useful thing to teach and I spend a huge amount of energy teaching this topic. Teaching the frequentist stuff has several advantages: the student starts to be able to differentiate BS from valid claims in the published literature (exaggerating only slightly: “we carried out a repeated measures ANOVA on this 2x2x7x14 design and found a marginally significant 18th order interaction (p=0.05921) that we therefore published in major journal X”). Most of the published stuff is like this; I routinely meet senior scientists from Ivy League unis who demand that I be impressed by the large number of (selectively published) significant effects on some phenomenon or the other. It’s important to be able to understand why one should remain skeptical and I wish these senior scientists had taken the time to study frequentist statistics. Another useful function of the frequentist methodology is the idea of hypothetical repeated sampling; understanding the properties of one’s design from this hypothetical sampling perspective is very valuable, and it is a good exercise to see this in action without the additional machinery (priors) that Bayes adds. Frequentist stats is a good gateway drug for Bayes. It is super useful in Bayes too; this is a great contribution of the frequentist way of thinking IMO. Finally, if one is not trained in frequentist methods, one will not understand 99.99% of the conference talks or papers one encounters.
I agree with Daniel fully that any scientist doing empirically driven work has to know linear algebra, calculus, logic (+programming) and the focus should be on mathematical modeling. However, the reality in the humanities (I’m thinking linguistics and psychology, and cognitive science more generally) is that this kind of training is not at hand when students come in for a degree.
My son is in Gymnasium (German high school), class 10, and his math textbooks are really sophisticated; they teach serious stuff in school here in Berlin, and the math and physics (hard science more generally) teachers are truly awesome. But quite a few of the students I get in undergrad can barely go past addition; they get frightened by the summation or the integral symbol. It’s even worse at the grad level; I get students from all over the world, and some of them claim to never have seen an algebra problem in their life and want to be taught fundamental stuff that is taught in class 9. They get overwhelmed if you ask them to review all this high school stuff themselves by googling it. Given these limitations, implementing Daniel’s ideas is simply impossible. I even give a Mindset lecture to my grad students to show them how to teach themselves stuff: https://youtu.be/317gI8WXSo4.
My solution was to have a Foundations of Math course at the grad level, covering linear algebra, pre-calc and calculus, probability theory. Then two courses (over a year) on frequentist statistics, and then two courses (over another year) in Bayesian stats. Students can doing Foundations of math parallel with the Intro frequentist class, so they can do five classes in two years and become functionally ready to do serious stuff in grad school. This is in theory. In practice, usually only a handful of people make it to my Bayes 2 class, there are many people failing even Frequentist Stats 1 (many complain in Freq Stats 1 that the math is too hard, even though I use only addition, subtraction, multiplication, and division). So there is a bottleneck in implementing Daniel’s ideas. I suspect the problem begins in school and is amplified in universities.
Incidentally, there is heavy resistance to my style of teaching statistics in my dept. I have been repeatedly advised to provide students with a cheat sheet with a if-else chart. If binary responses do this test, if continuous responses, do a t-test, if a Likert scale just close your eyes and do a t-test, if there is too little data to do a t-test do a non-parametric test, etc. Once I retire (9 more years) I think that is again how stats will be taught in Potsdam.
What I want to say here is that I tried to approximate the solution Daniel proposes, but with very dissatisfying results that seem to have systemic causes going back to high school and earlier stages of education.
I’m not so sure there’s a mathematical bottleneck preventing social scientists from becoming proficient in statistics, any more than there’s a mathematical bottleneck in their becoming proficient in neuroscience or pharmacology. Neuroscience is hard, you have to follow a pretty rigorous academic path to become proficient in it, and even though it would be helpful to understand, most people who do social science don’t also do neuroscience. If we take seriously Andrew’s mantra that statistics is hard–and I do–should we expect very many people to become proficient in it, no matter how useful it may be? In other words, why should the kind of person who is interested in studying, say, cognitive behavioral therapy, and who has an aptitude for studying psychological theory, also be the kind of person who is interested in and has an aptitude for statistics? Maybe someone studying therapy should consult with neuroscientists for neuroscientific expertise, and with statisticians for statistical expertise.
Well, if you are using statistics in neuroscience you’d better be proficient in it. You may still want to consult a statistician, but you won’t even understand what they are advising you to do unless you already understand statistics well. Just read pretty much any neuroscience paper to see what happens if you don’t know what you are doing with the statistical tests reported there. fMRI studies are especially fun.
What’s so hard about neuroscience or pharmacology in particular?
Shravan, imagine if we let people become civil engineers without really studying Newtons mechanics, or doctors without studying physiology or anatomy.
You just can’t be a quantitative researcher without having a minimum of high school algebra, differential and integral calculus, a semester of some kind of probability, a programming course, and then some kind of class involving dynamics (ie. Things that change through time, could be ODEs or iterated functions or computational/rule based = agent based models). Then when you take a “statistics” class it should be a class in using those tools to build models and then looking at how to infer the unknowns from the data. The second stats course should involve analyzing how to get precision from fiddling with the experimental design, and be largely simulation based.
Im not arguing for a proofs based approach that a math major would get, or a high end CS theory approach to computing or anything like that. But anyone afraid of the quadratic formula or a summation sign just can’t be a quantitative researcher.
The world would be a better place if we stopped pretending otherwise.
While more math knowledge is always good, I don’t think that’s generally the problem in much of the social sciences. The problem is that people are trying to apply this math to crudely formulated problems with more noise than signal and pretending that the estimated parameters are proof of a signal. There are entire fields where the measurements are nowhere near where they need to be to build a scientific body of knowledge given current technology and the price of it — but, we continue to churn out nonsense and publish it using these methods, pretending that the measurements are meaningful.
ODEs will not solve the problem of crude measurement.
Yes, Daniel, we should raise our expectations to the appropriate level. But here in the thick of things, I can’t see any way to do that, specifically in my world. On the plus side, every few years I get through to some students and they go on to do serious work. But I am only relating my experience here.
Curious makes a good point, about crude, noisy measurement and vague hand-waving theories being tested using oversimplified linear (mixed) models. Stan offers some entries into the universe of more detailed model specification, but the noisy measurement problem is a killer one with no clear/easy solution.
Shravan, it’s a problem of group action. We can’t as individuals fix the world. What is needed is a political movement.
Curious, it’s not just math knowledge that I’m arguing for, but rather mathematical modeling knowledge. There are tons of sophisticated mathematicians who study, topology or abstract algebra or probability theory etc etc, who I wouldn’t trust to solve a simple 1st year textbook problem of how much force does it take if you lift such and such engine with such and such pulley system.
We want people who can look at their issue in whatever it is, sociology, ecology, atmospheric physics, economics, human rights abuses, voting patterns… and think to themselves “such and such is probably going on, I could call that X and then there’s some function of the number of people involved N, such that f(N) – X is a time-varying value … etc etc etc” and arrive at an algebraic equation which they could then rearrange and decide “here are the quantities we could measure and here are the unknowns and if we collect this data, we can find out approximately the values of the unknowns through Bayesian inference”
But instead people don’t even try to do the science, like with the coal near the river in china problem, or any of those examples Andrew’s posted for decades. Show me even one that thinks even for a few minutes about the mechanisms or regularities of the process. Conservation laws, or changes through time, or stocks and flows… anything.
Daniel said,
“I’m not arguing for a proofs based approach that a math major would get, or a high end CS theory approach to computing or anything like that. But anyone afraid of the quadratic formula or a summation sign just can’t be a quantitative researcher.
The world would be a better place if we stopped pretending otherwise.”
I think that you’re missing another important thing here — what I would call “qualitative analysis” –it’s something I encountered as an undergraduate in both calculus and in a mechanics course in physics. What I mean by “qualitative analysis” is deducing qualitative information about something like the trajectory of an object, by analyzing information about how the object is moving.
For example, if we define f(t) to be the height of the object at time t, we might be able to deduce (for example, from a differential equation deduced from laws of physics plus constraints) that df/dt is proportional to t-cubed. From this we can deduce that f is increasing when t >0, and decreasing when t < 0. Using this, we can give a rough sketch of the trajectory.
I don’t necessarily disagree with you, but I think you’re underestimating how much time would be involved in acquiring these skills (unless you’re reducing the actual subject-matter content to a footnote). Moreveover, the analogy to doctors and civil engineers doesn’t really work. In my experience, students in say, linguistics, don’t start out wanting to quantitative researchers. They are interested in language. Moreover, what they know about “language” from high school is often either irrelevant or has to be unlearned. So they have to learn from the ground up phonetics, phonology, semantics, syntax, and morphology (these are what is comparable to someone wanting to be a doctor having to lean anatomy and physiology). And those are just basics, comparable to arithmetic. There’s a number of different specializations within linguistics that any competent researcher would have to learn as well, each with its own body of previous research, theoretical presuppositions, and distinct methodologies. It’s usually only after several years of study that a subset of students realize that they want to do quantitative research and/or computational methods. So then there’s all that to learn as well. All of this in an age where people are calling for students to graduate in a shorter amount of time, an increasing number of subjects that they are supposed to know, and, for graduate students, publication expectations.
I agree that the problem is that if students/researchers don’t know these things they shouldn’t be doing that kind of research in the first place. Still, saying that world shouldn’t be a better place that it is doesn’t help someone who actually has to work in the one that exists.
Joe, I do agree that it is a lot of work to catch up on the quantitative side of things. But I never understood this argument about lack of time. People often tell me they lack the time to do this or that, especially my students. I studied statistics over four years while on hemodialysis (I mean literally; I did the homework while connected to the dialysis machine). Currently I work at half my capacity because I am forced to go to day time dialysis (instead of night dialysis) after the pandemic started, which costs me 20 hours a week. I still manage to get stuff done. If I were not on dialysis, I could do a lot more.
When I hear such arguments, that one wants to do quantitative work but practically speaking one doesn’t have the time to acquire the necessary knowledge for it, I feel that the person needs to take a serious look at their day to day life. I suspect this is a lack of willingness to give up on other things (facebook, twitter, Instagram, etc. etc.). That is fine of course; these and other things must add value otherwise one wouldn’t spend time on them.
But one can’t invoke lack of time; it’s lack of motivation. A fruther unfortunate fact is that one can just completely get away with knowing nothing when one does a statistical analysis in linguistics. One can become a psycholinguist in some top Ivy League uni or the like, with 30 years behind one and an h-index of 60 or 80 or whatever, and still not be able to do a one sample t-test correctly, but this doesn’t stop one from publishing hundreds of experimentally oriented papers in top journals like PNAS and JML and Cognition. In an alternative universe when nonsensical analyses would be rejected, people would step up and just learn the needed material because it would then be an existential threat.
Also, as you say, linguistics is a huge area and you can just choose to work in sub-areas where no quantitative knowledge is needed. I think that’s just fine. One can have a really productive life doing head driven phrase structure grammar or categorial grammar or (gulp) minimalism or whatever it is called now. Why not? It all has value. Intuition-based linguistics has brought us very far and I think that all those papers trashing intuition and demanding a quantitative turn are misguided. Just do something else that falls within your skill set (that’s what I do too, stay in my lane :).
@Daniel: !We want people who can look at their issue in whatever it is, sociology, ecology, atmospheric physics, economics, human rights abuses, voting patterns… and think to themselves “such and such is probably going on, I could call that X and then there’s some function of the number of people involved N, such that f(N) – X is a time-varying value … etc ” makes you sound a bit like a system dynamicist. See the description and ToC to John Sterman’s /Business Dynamics/ (https://jsterman.scripts.mit.edu/Business_Dynamics.html) for a bit of what one major textbook in that field covers.
Some have described the domain of SD work to be heavily problem solving in “managed systems”: systems that have people (policy makers) in the loop, such that some pieces are more “physics” and other pieces are more human behavior.
I very much agree with Daniel’s sentiment re: mathematical modelling. It’s practically scandalous how we generally do not teach principles of mathematical modelling, and basic philosophy of science, in many so-called “math based” or “scientific based” undergraduate programs.
This is not nearly limited to the social sciences. I’m a professional structural engineer whose undergraduate education was at a university ranked as a top 10 research institution in the world for civil engineering* (at the time), and I cannot recall a single moment where any design approach was referred to as being a mathematical model (which they clearly are). I certainly had no training in how proper scientists – in the spirit of Meehl and company – evaluate and appraise theories; the closest I had was testing a concrete beam to failure in the lab, which was cool, but not nearly enough.
This state of affairs regrettably allows practicing engineers to go about designing structures without having the tools required to check their understanding of the world except via peer review; which, as we know, is no surety. The problem is one of multiple hypotheses (one knows how to design; one is really a poor designer) mapping to the same outcome (the building stands) due to safety factors and the lack of seeing a true design level event. We even have data where design level events were recorded – such as certain earthquakes in New Zealand – and the damage done was vastly in excess of that expected if those buildings were designed/built to the code standards of the day. The conclusion was that the buildings were under designed and now calculations are required as part of submittals to the AHJ prior to issuing a permit (not a universal practice by any means).
The point I’m trying to drive home is that we need an increased focus on model building (mathematical, mental, etc) and on how we ought to evaluate those models in pretty much all disciplines (even non-scientific ones). The world would be much better off if this was the case. Oh also live in fear the next time you set foot in a high-rise building!
*Having been properly educated by this blog I certainly know these rankings are a fool’s errand, but it adds street cred to what I’m saying in this instance, so….I’m including it, ha!
Bill, I’ve often wanted to read something on System Dynamics so I could have a better sense of how to communicate with those people. I am pretty sure I am already in the same camp, but not necessarily in the same jargon space ;-).
I bought Donella Meadows’ “Thinking In Systems” because it was highly recommended in a thread on the Julia Discourse forum. We’ll see what comes of it.
It’s far more important to teach insight instead of blindly following routines than what approach is taught. Whoever thinks that Bayesian statistics won’t be misused and misinterpreted when taught in the cookbook/cheat sheet/get results quickly manner that is behind the misuses of NHST is deluded.
Yes, the abuse of Bayesian statistics is well under way. As soon as one makes a cookbook up, the story is already over.
I agree. Bayesian is the new ‘Ketogenic’ diet, solving all the ills that plague us. Until it isn’t, that is.
I read the blog post and also did not find it upsetting or wrong. The issue that Henke raises about taking a single stats course I also don’t agree with. Of course, people should take more – but one good (emphasis on that word) stats course should be enough for someone to get started and accomplish some analysis of their own as well as being able to critique what they read. As Daniel suggests, many/most stat courses are not good. I personally think it need not be Bayesian (disclosure: I was not trained in Bayes), but NHST is simply a huge devotion of time wasted (or perhaps of negative value) in the traditional stats course that people take. If you drop the tedious approach that is used in most undergraduate texts (single variable, null hypothesis for the mean, building up (!) to 2 variable t tests,…. ending with ANOVA or perhaps a simple multiple regression), quite a lot can be accomplished in that first stats course. And, if we are ambitious with what can be done, I think it leads to a healthy humility and skepticism about what it take to do good statistical work.
The problem with undergraduate statistics education is the statistics.
If you learn to perform a bunch of fancy calculations with cool looking distributions that promise gold at the end of the computation, you are going to press go and collect $200 all day long. But if you are taught to actually look at evidence in relation to a hypothesis in the way that a proper scientist does you’ll look at much methods with a much more appropriate level of skepticism.
This is a philosophical problem and not a mathematical one in my mind. Until we ground ourselves with the proper philosophical vantagepoint it doesn’t matter what statistics one learns (frequentist, objective Bayesian, subjective Bayesian, whatever), it will all lead to the same morass we are in now. The only thing Bayesian statistics has going for it (in and of itself) over frequentist stats as a first introduction is that the philosophical component has to be broached at least in a modest way.
This need not take a 4-year degree to complete but rather a single course with appropriate focus. I mean Meehl’s lecture series – available online for free – is more than enough to get a proper footing.
Sorry I didn’t quite explicate that I am agreeing with you wholeheartedly (in case it’s not clear)!
To complement what Daniel and Dale wrote, the important thing is to understand the models that underlie the procedures that you use. Thinking Bayesianly, i.e., model your unknowns and condition on the data, is also important. If you do those things, you don’t need a degree in statistics.
“My reading of the widespread misuse of stats in social science is that it is mostly caused by people – like me – who just don’t know enough to responsibly apply the methodologies and analytic techniques that they are able to mechanically execute.”
I think the issue is more of the power structure and culture surrounding the publication process, academia, and other areas of research and data science:
Why is it not ok to be able to make mistakes and learn from them (disclaimer: obviously the consequences of mistakes would vary across the topic area and the outcome)?
Why isn’t transparency in the data and analysis not the standard so we can more easily catch mistakes?
Why does tenure and merit promote publication requirements that pressure researchers and incentivizes more mistakes in research?
Why are journal reviewers not properly compensated so that we can have better reviews to catch mistakes before publication?
Why do we see having a publication revised or taken down as a black mark on a record instead of as a sign that someone values the impact their research can have and acts accordingly.
Why do we continuously see smart and skilled professionals engaging in unethical research practices?
Why do journals fight against criticism or treat criticisms as another avenue for publications?
Why do we have “standards” for methodology and inference that should probably be an option, rather than the “right and accepted” method?
Why do we use novelty as a publication requirement when we have a replication crisis?
Why do we often not treat people that make mistakes as humans instead of treating them like they’re idiots for not knowing everything?
Why do we treat statistics as just a stamp or seal of approval to do what we already want to do, rather than as a method for understanding, evaluating uncertainty, and making decisions?
These to me will always be the most impactful because they are barriers to learning and doing what’s right. Methodology, research, statistics, data science, measurement, etc… are all skills that are meant to be applied and everyone has a right to try their hand at it and learn. I don’t want to be a teacher at all, but I feel awesome when I see someone learn about statistics and are engaged with the field. But from my perspective, I see the above as impacting knowledgeable and skilled professionals’ confidence and work, leading to misuse and mistakes. But of course I have my own biases on this matter and I probably lost the thread somewhere around 25% through this post.
+ 1
“phdummy”,
I appreciate your comments. My formal education has been mostly as a mathematician, but I got interested in statistics as a consequence of my interest in improving the education of secondary math teachers, when AP statistics became common in high schools. Once I had dipped my toe into statistics a bit, I was curious to learn more. I sat in on one of the graduate statistics courses, and then asked the senior statistician in my department if it might be possible for me to teach a graduate course in statistics some time — to my amazement, he said, “How about teaching graduate regression analysis this fall?” (As it happened, the department wasn’t very enthusiastic about statistics, but had “supported” the small statistics group by an arrangement that each year they would hire an instructor in the field of “pure math” probability to teach two of the statistics courses required for the master’s in statistics program.) Probably based on the questions I had asked him as I prepared and taught the new stats course for teachers, he thought I would do a better job of teaching regression than the instructors who had been teaching it as a source of their paycheck rather than as their professional interest. Also, at this time, NSF was offering summer courses for mathematicians branching out to statistics, and I signed up for two or three of them.
So after a couple of years, I ended up teaching graduate ANOVA as well as regression and the teacher prep course, and enjoying them all.
After a couple more years, I had gained a lot of experience and had become aware of how often statistical concepts are misunderstood. By that time, the newly (slightly) expanded statistics group here was starting a summer “continuing education” program in statistics, so I submitted a proposal for a course titled: “Common Mistakes in Using Statistics: Spotting Them and Avoiding Them.” It was approved, brought in a large enrollment, and continued for several years. As I got more involved in other duties, I passed the summer course on to another colleague whom I though would do a good job. But I have posted the notes and other relevant information for the course at https://web.ma.utexas.edu/users/mks/CommonMistakes2016/commonmistakeshome2016.html . I think you might find at least some parts of them interesting and helpful.
I completely agree that everyone should be able to learn science by doing science, and that our institutions should embrace this philosophy. But I also think it’s only fair to tell people that the institutional philosophy is actually as you describe it, full of incentives to pretend–or to actually believe–that one’s published work is a demonstration not of a curious mind learning but of an authority teaching. In terms of statistics, this means telling people up front that doing “real” statistics as a non-statistician leaves their work open to criticism and, potentially, immediate invalidation by actual statisticians. And, consequently, all the incentives will line up to do “safe” statistics–whatever everyone else in their field is doing–instead, because at least then your peers will have your back.
“My reading of the widespread misuse of stats in social science is that it is mostly caused by people – like me – who just don’t know enough to responsibly apply the methodologies and analytic techniques that they are able to mechanically execute.”
I think the issue is more of the power structure and culture surrounding the publication process, academia, and other areas of research and data science:
Why is it not ok to be able to make mistakes and learn from them (disclaimer: obviously the consequences of mistakes would vary across the topic area and the outcome)?
Why isn’t transparency in the data and analysis not the standard so we can more easily catch mistakes?
Why does tenure and merit promote publication requirements that pressure researchers and incentivizes more mistakes in research?
Why are journal reviewers not properly compensated so that we can have better reviews to catch mistakes before publication?
Why do we see having a publication revised or taken down as a black mark on a record instead of as a sign that someone values the impact their research can have and acts accordingly.
Why do we continuously see smart and skilled professionals engaging in unethical research practices?
Why do journals fight against criticism or treat criticisms as another avenue for publications?
Why do we have “standards” for methodology and inference that should probably be an option, rather than the “right and accepted” method?
Why do we use novelty as a publication requirement when we have a replication crisis?
Why do we often not treat people that make mistakes as humans instead of treating them like they’re idiots for not knowing everything?
Why do we treat statistics as just a stamp or seal of approval to do what we already want to do, rather than as a method for understanding, evaluating uncertainty, and making decisions?
These to me will always be the most impactful because they are barriers to learning and doing what’s right. Methodology, research, statistics, data science, measurement, etc… are all skills that are meant to be applied and everyone has a right to try their hand at it and learn. I don’t want to be a teacher at all, but I feel awesome when I see someone learn about statistics and are engaged with the field. But from my perspective, I see the above as impacting knowledgeable and skilled professionals’ confidence and work, leading to misuse and mistakes. But of course I have my own biases on this matter and probably lost the thread somewhere along the beginning of the post lol.
I agree that NHST is a bit of a distraction in first stats course but I wonder how it could be practically phased out. If you were to, right now, design an introductory statistics course that did not include NHST I would think it would not be very effective. In the sense that students would go out into the real world where all their employers, advisors, etc. are asking them to do NHST and the students would have no idea what to do. Nor would they have the expertise to explain why they think that’s the wrong approach and propose an alternative. So for those proposing dropping NHST, I wonder what you think the most practical approach would be?
Getting back more to the original post, Andrew makes a really good point about the balance between having enough confidence to start while avoiding over-confidence. I find this really difficult! But I guess the way is to go forward and try but acknowledge that you will probably make mistakes. And if you’re e.g. doing research you have to go forward hoping that your mistakes will either be inconsequential or caught by co-authors, peer review, or post-publication review.
Good points. (You might find my comments to phdummy, and the link in them, interesting and/or informative.)
Teach the same math but change the “null hypothesis” to something predicted by a theory.
“So for those proposing dropping NHST, I wonder what you think the most practical approach would be?” Good question. I think I’ve seen a couple of paths, but they don’t span the entire career space.
No. 1: Pick a field where other non-NHSTers have begun to make inroads already, and build on that. Or pick a field that hasn’t benefited much from statistics yet, and just do it the way you think right.
No. 2: Become at least a part-time methods person, and publish papers and books, teach, and present on how to address the needs of field X without even mentioning NHST.
For an example of the second, I think one can look at Harry L. Van Trees’ Detection, Estimation, and Modulation Theory, Volume 1 (1968). That first edition, as I recall, covered detection and estimation theory from a Bayesian perspective without mentioning any alternatives. All the calculations were done analytically, making it more challenging and limiting what could be done practically. It was taught after a course in probability using Dubes Theory of Applied Probability.
I didn’t go into that field, but reading Amazon’s description and reviews of the 2013 second edition makes it seem as if DEMT set the standard for 40 some years of work in this field, only to be supplanted by a much thicker second edition that apparently now includes more computerized approaches.
Imagine if the author of that post had started out by giving the same advice–that a couple of classes on statistics are a good start, and you can get even over time better if they continue reading, googling, practicing, etc.–but she went on to write the other things those learning statistics need to hear. That learning more about statistics will quickly lead them to realize that the kinds of analyses everyone else in their lab/field is doing, including their advisor/supervisor, are wrong. That, ethically, they’re obligated to take their names off any papers with a p-hacked NHST. That papers they write without p < .05, and with appropriately cautious conclusions, are unlikely to be published in top journals, or to serve as the basis for successful funding proposals to many institutions. That if they can't get funding for studies with large n's and reliable measures, they'll have to admit in their papers (and in future proposals) that the information content in their results is minimal. That they will not for the first several years of their career (at least) actually know if the research designs and statistical analyses in much of their published work are correct, because peer reviewers don't know, either.
Having imagined all that, you surely can understand Henke's objection. He's not saying the author's advice is wrong, he's saying she's giving the wrong advice. Doing research with sound, ethical statistics isn't just technically hard, it's also hard because it means swimming against a powerful paradigmatic current. If people aren't getting this advice, even from statisticians, is it any wonder they're shocked and defensive when this blog offers accurate criticism of their work? In short, how can we teach people the worth of good statistics if we don't also teach them its cost?
Michael Nelson wrote:
“Imagine if the author of that post had started out by giving the same advice–that a couple of classes on statistics are a good start, and you can get even over time better if they continue reading, googling, practicing, etc.–but she went on to write the other things those learning statistics need to hear. That learning more about statistics will quickly lead them to realize that the kinds of analyses everyone else in their lab/field is doing, including their advisor/supervisor, are wrong. That, ethically, they’re obligated to take their names off any papers with a p-hacked NHST. That papers they write without p < .05, and with appropriately cautious conclusions, are unlikely to be published in top journals, or to serve as the basis for successful funding proposals to many institutions. That if they can't get funding for studies with large n's and reliable measures, they'll have to admit in their papers (and in future proposals) that the information content in their results is minimal. That they will not for the first several years of their career (at least) actually know if the research designs and statistical analyses in much of their published work are correct, because peer reviewers don't know, either."
This reminds me of an incident in a Ph.D. dissertation exam for a biology Ph.D. student who had taken a couple of graduate statistics courses from me. Her advisor didn't seem to be aware of "the problem of multiple inference on the same data set." But when the candidate pointed out the problem to her advisor, and I and another member of the committee supported her, the advisor was duly chagrined — and did listen to our careful explanations of what the problem was.
Daniel, Donella Meadows has done some good, famous modeling, and she writes well, too (browse https://donellameadows.org/). That’s not a bad place to start.
If you have a second sometime after you finish it, I’d be curious in your reaction.