“Study finds ‘Growth Mindset’ intervention taking less than an hour raises grades for ninth graders”

I received this press release in the mail:

Study finds ‘Growth Mindset’ intervention taking less than an hour raises grades for ninth graders

Intervention is first to show national applicability, breaks new methodological ground

– Study finds low-cost, online growth mindset program taking less than an hour can improve ninth graders’ academic achievement
– The program can be used for free in high schools around U.S. and Canada
– Researchers developed rigorous new study design that can help identify who could benefit most from intervention and under which social contexts

A groundbreaking study of more than 12,000 ninth grade U.S. students has revealed how a brief, low-cost, online program that takes less than an hour to complete can help students develop a growth mindset and improve their academic achievement. A growth mindset is the belief that a person’s intellectual abilities are not fixed and can be further developed.

Published in the journal Nature on August 7, the nationally representative study showed that both lower- and higher-achieving students benefited from the program. Lower-achieving students had significantly higher grades in ninth grade, on average, and both lower- and higher-achieving students were more likely to enroll in more challenging math courses their sophomore year. The program increased achievement as much as, and in some cases more than, previously evaluated, larger-scale education interventions costing far more and taking far longer to complete. . . .

The National Study of Learning Mindsets is as notable for its methodology to investigate the differences, or heterogeneity, in treatment effects . . . the first time an experimental study in education or social psychology has used a random, nationally representative sample—rather than a convenience sample . . .

Past studies have shown mixed effects for growth mindset interventions, with some showing small effects and others showing larger ones.

“These mixed findings result from both differences in the types of interventions, as well as from not using nationally representative samples in ways that rule out other competing hypotheses,” [statistician Elizabeth] Tipton said. . . .

The researchers hypothesized that the effects of the mindset growth intervention would be stronger for some types of schools and students than others and designed a rigorous study that could test for such differences. Though the overall effect might be small when looking at all schools, particular types of schools, such as those performing in the bottom 75% of academic achievement, showed larger effects from the intervention.

More here.

I’m often skeptical about studies that appear in the tabloids and get promoted via press release, and I guess I’m skeptical here too—but I know a lot of the people involved in this one, and I think they know what they’re doing. Also I think I helped out in the design of this study, so it’s not like I’m a neutral observer here.

One thing that does bother me is all the p-values in the paper and, in general, the reliance on classical analysis. Given that the goal of this research is to recognize variation in treatment effects, I think it should be reasonable to expect lots of the important aspects of the model to not be estimated very precisely from data (remember 16). So I’m thinking that, instead of strewing the text with p-values, there should be a better way to summarize inferences for interactions. Along similar lines, I’m guessing they could do better using Bayesian multilevel analysis to partially pool estimated interactions toward zero, rather than simple data comparisons which will be noisy. I recognize that many people consider classical analysis to be safer or more conservative, but statistical significance thresholding can just add noise; I think it’s partial pooling that will give results that are more stable and more likely to stand up under replication. This is not to say that I think the conclusions in the article are wrong; also, just at the level of the statistics, I think by far the most important issues are those identified by Tipton in the above-linked press release. I just think there’s more that can be done. Later on in the article they do include multilevel models, and so maybe it’s just that I’d like to see those analyses, including non-statistically-significant results, more fully incorporated into the discussion.

It appears the data and code are available here, so other people can do their own analyses, perhaps using multilevel modeling and graphical displays of grids of comparisons (but not grids of p-values; see discussion here) get a clearer picture of what can be learn from the data.

In any case, this topic is potentially very important—a effective intervention lasting an hour—so I’m glad that top statisticians and education researchers are working on it. Here’s how Yeager et al. conclude:

The combined importance of belief change and school environments in our study underscores the need for interdisciplinary research to understand the numerous influences on adolescents’ developmental trajectories.

P.S. More here from Jared Murray, one of the authors of this study.

56 thoughts on ““Study finds ‘Growth Mindset’ intervention taking less than an hour raises grades for ninth graders”

  1. I checked the pre-registration, and this was written somewhere at the end:

    “This plan received input from these individuals:

    Andy Gelman (Columbia), Jordan Axt (Virginia), Todd Rogers (Harvard), Mike Weiss (MDRC).”

    Does anybody know who this “Andy Gelman” from Columbia is?

    And why was he even asked, is he any good at this statistics stuff?

    • It will be interesting for me to see the one hour Growth Mindset Intervention administered to the 9th graders. If it has demonstrated efficacy> great.

      All in all, I favor a developmental [Deweyian] model of education, requiring a real knack for understanding the learning needs of individual kids.

  2. I was wondering whether this “growth mindset” could be due to having more motivation or students trying their best more. In light of this, i went on a little search to find the materials. In the Nature article, there is a link to “Supplementary information section 4” which contains some material information if i am not mistaken.

    There i can see what looks like screenshots of the different conditions.

    # In figure 4.2.2. i can read among other things: “why do scientists say that the harder you work, the smarter your brain becomes?”
    # In figure 4.2.3 i can read among other things: “What are one or two challenging things you are using your brain to learn in high school classes?”
    # In figure 4.2.4 i can read among other things: “It’s not just about effort: Use the right strategies” & “Sometimes people want to learn something challenging, and then they try hard”
    # Additionally, if i understood things correctly, the term “growth mindset” is NOT being used throughout the intervention but instead the term “learning mindset” is being used.

    I tried to find out what is exactly meant by “growth mindset”. The following is from the paper:

    “The specific intervention evaluated here—a growth mindset of intelligence intervention—addresses the beliefs of adolescents about the nature of intelligence, leading students to see intellectual abilities not as fixed but as capable of growth in response to dedicated effort, trying new strategies and seeking help when appropriate”

    Now, i am wondering if “growth mindset” is a term that is being used in a way that is directly related to intelligence. And if so, it seems to me that the intervention seems to not talk about intelligence very much, if at all. It’s about how the brain is like a muscle, and how you can try new strategies if something doesn’t work, etc. I am not sure if that’s the same thing as “intelligence”.

    Or to use the word of the paper itself: “The growth mindset intervention communicates a memorable metaphor: that the brain is like a muscle that grows stronger and smarter when it undergoes rigorous learning experiences”. And “The intervention can lead to sustained academic improvement through self-reinforcing cycles of motivation and learning-oriented behaviour.”

    I am wondering whether the effect might be due to things like being more motivated, eager to learn, wanting to do their best, etc. After writing the above, i just came across the following which is exactly in line with my points above. The authors themselves even ponder about “effort”, and write the following:

    “Among these features, our intervention mentioned effort as one means to develop intellectual ability. Although we cannot isolate the effect of the growth mindset message from a message about effort alone, it is unlikely that the mere mention of effort to high school students would be sufficient to increase grades and challenge seeking”

    I think the study should have been designed to at least try and tackle the effort-growth mindset thing. That just seems rather obvious to be a possible confound, and to try and discern from the effect you are primarily looking at.

    Overall, i guess my question, and possible problematic issue, is that i wonder if phrasing “growth mindset” as something related to “intelligence” could be a (unnessasary?) sub-optimal way to phrase and/or talk about this all. Perhaps the “intelligence” (whatever that means) does not change at all, but students with a “growth mindset” maximize their potential more, or something like that.

    • Quote from above: “I think the study should have been designed to at least try and tackle the effort-growth mindset thing. That just seems rather obvious to be a possible confound, and to try and discern from the effect you are primarily looking at.”

      The control condition seems to be about learning about the brain, more specifically some physiological features and things like that.

      It seems to me that the “groth mindset” condition could easily remove all the “effort” stuff and simply only state facts (or “facts”) concerning the malleability, and the brain is a muscle, kind of stuff (just like the control condition simply only stated facts about the brain).

      In my view, that would be a more “fair” comparison, and would be a step towards trying to separate variables like “effort” from the effect you want to investigate.

    • Well I see your last point. But as I remember from several Linkedin discussions, this emphasis [growth mindset] was to counter the overemphasis on IQ scoring and genetics.

      I myself am very sympathetic to any effort to build childrens’ confidence & motivation. Adults can enable stigmatization of childrens’ abilities. For example I abhor the designation ‘At Risk’ children. Parents can cue their kids to feeling insecure. Dumping their own anxieties on their children, which can affect their learning in school.

      In the current crises of knowledge environment, we should all consider ourselves at ‘At Risk’ for being bamboozled by some of the claims made

      • Quote from above: “I myself am very sympathetic to any effort to build childrens’ confidence & motivation.”

        I am as well!

        I just notice that 1) it is not clear to me what this “growth mindset” is all about (e.g. in relation to “intelligence”), and 2) we’re talking about science here, so we should always stick to facts, logic, etc.

        For instance, it’s not “okay” in my book to tell children stories in schools via interventions (or whatever) that are not true even if that could lead to “positive” things (or things that you think are “positive”).

        It’s also not “okay” in my book to use interventions that are really about X, to try and get a message about Y across to schools, the public. the media, etc.

        • Suppose you tell children “you should never eat mushrooms that you don’t buy at a store because they’re poisonous and can kill you”. Suppose that this prevents 1000 child deaths per year.

          Later, these children grow up and find out that there are actually perfectly fine wild mushrooms if you have a lot of knowledge to identify them properly, like chanterelles and boletes and truffles….

          is this a terrible thing?

          sometimes we give simple answers to complex questions because they are useful heuristics that do good things, but they aren’t strictly true… I don’t have a problem with this provided we don’t actively censor the truth so people can’t find it out.

        • Quote from above: “sometimes we give simple answers to complex questions because they are useful heuristics that do good things, but they aren’t strictly true… I don’t have a problem with this provided we don’t actively censor the truth so people can’t find it out.”

          I think i understand your point.

          I am in favour of trying to be as precise as possible in things, and like you mentioned to definitely NOT actively censor the truth (leaving aside whether there is a “truth” and/or whether we can know it). To use your example, it think it would be more “correct”, and just as easy, to state something like:

          “you should never eat mushrooms that you don’t buy at a store because they COULD BE poisonous and can kill you.”

          (And you could then even add something like: “Mushrooms that you buy at a store are checked and safe to eat, so you should only eat mushrooms that were bought in a store”).

          Concerning the “growth mindset” study: when reading some of the things in the intervention (e.g. “when you work hard to learn something new -like a new type of math problem- the connections in your brain get stronger” & “the brain can get stronger at any age, but there are two times in life where the brain is especially ready to grow”) i was wondering whether this is actually correct or not.

          Then i began to wonder if it would be “okay” to tell these things if they were NOT true but could have a “positive” effect. I think that would NOT be okay.

          I think this view may also (partly) have to do with my philosophy on life ,and things, and stuff. I don’t like to steer and nudge people, whatever age they are. I think am mostly about the individual, and for taking and giving responsibility, and for freedom of choices, etc.

          I also think it’s perfectly fine if some people are more “intelligent” (whatever that means, or however that’s measured) than other people. I also think it’s perfectly fine if some people do not have “good grades” or want to try their best in schools.

          This is all coming from someone who:

          # always had “excellent” grades as a child
          # then lost interest in highschool with regards to “learning”, and the schoolsystem
          # subsequently dropped out of high-school at 16/17 (if i remember correctly)
          # did some cr@ppy manual labour-type jobs for a decade
          # did some traveling (Europe by train, and some long distance cycling)
          # had a nervous breakdown, but recovered
          # then went back to university to try and “give it my all in a final attempt”
          # then wondered who the h#ck these people were that looked like managers to me, but were apparently “scientists”
          # then graduated
          # then began to wonder why my education has been so bad there
          # then began to wonder what i could even do with my university diploma
          # then began to wonder if the only use of my unversity degree could be as a piece of (expensive!) toilet-paper
          # then began to wonder if i should have just stayed employed, and try and “work my way up”
          # then started to comment on some statistical & social science blog to “try and use my knowledge for something “useful” in a final attempt”
          # then started to wonder why i was even doing that
          # then started to comment on said statistical & social science blog in a meta-comment kind of way that possibly even more clearly showed how i truly should have done things differently in my life

        • Quote from above: “What is your definition of ‘intelligence’?”

          I don’t have any i think. I think i view it mostly in relation to an IQ-test. But with the note that these tests probably don’t measure all kinds of “intelligence”, and that these tests may measure things sub-optimally, etc.

          I don’t care very much about “intelligence”. I don’t care if i am “intelligent” or not. I even think it may not be a “good” thing for people to be very “intelligent” for instance.

          Is “intelligent” the same as being “smart”? Is “intelligent” the same as being “wise”? If not, i think maybe “intelligent” is less important than being “smart” and/or “wise”. And maybe being “smart” and/or “wise” is doing and being something that makes you happy and makes you “you”.

        • Quote from above: “Is “intelligent” the same as being “smart”? Is “intelligent” the same as being “wise”? If not, i think maybe “intelligent” is less important than being “smart” and/or “wise”. And maybe being “smart” and/or “wise” is doing and being something that makes you happy and makes you “you”.”

          I like (some) movies, and (some) series, and (some) music. I like to sometimes combine it with things i think of and write. I thought about the following two things as a result of the discussion here:

          1) (A scene from) one of my favorite movies “Good Will Hunting”:

          https://www.youtube.com/watch?v=mJHvSp9AKYg

          2) The following lyrics of the song “Wonderful World” by Sam Cooke

          “Don’t know much about history
          Don’t know much biology
          Don’t know much about a science book,
          Don’t know much about the french I took
          But I do know that I love you,
          And I know that if you love me, too,
          What a wonderful world this would be

          Don’t know much about geography,
          Don’t know much trigonometry
          Don’t know much about algebra,
          Don’t know what a slide rule is for
          But I do know that one and one is two,
          And if this one could be with you,
          What a wonderful world this would be

          Now, I don’t claim to be an “A” student,
          But I’m tryin’ to be
          For maybe by being an “A” student, baby,
          I can win your love for me

          Don’t know much about history,
          Don’t know much biology
          Don’t know much about a science book,
          Don’t know much about the french I took
          But I do know that I love you,
          And I know that if you love me, too,
          What a wonderful world this would be

          History
          Biology
          Science book
          French I took
          But I do know that I love you,
          And I know that if you love me, too,
          What a wonderful world this would be”

      • Quote from above: “I myself am very sympathetic to any effort to build childrens’ confidence & motivation.

        It’s perhaps also interseting, and/or useful, to think about things like confidence and motivation. It may not necessarily be the case that the “growth mindset” stuff actually enhances children’s confidence and motivation. Or perhaps it only temporary increases confidence and motivation, but then in a few years time the kids get double or triple the disillusion, and loss of motivation, and dents in their confidence, when things get even harder and effort alone is not enough or way too much effort is needed to keep up.

        I don’t know, and i am pretty sure the “growth mindset” people also don’t know!

        Perhaps (true) confidence and motivation comes from:

        # failing
        # from learning/knowing where your limits are
        # from learning/knowing that not everybody is the same
        # from learning/knwoing that you CAN’T do everything you want
        # from learning/knowing what you want to do
        # from learning/knowing what you enjoy
        # from learning/knowing what you are good at

      • Anon:

        An effect of 0.1 GPA point is a lot. One way to think about this is that it’s equivalent to a treatment that raises GPA by 1 point for 10% of people and has no effect on the other 90%. That’s a bit of an oversimplification, but the point is that this sort of intervention might well have little or no effect on most people. In education and other fields, we try lots of things to try to help students, with the understanding that any particular thing we try will not make a difference most of the time. If mindset intervention can make a difference for 10% of students, that’s a big deal. It would be naive to think that it would make a difference for everybody: after all, many students have a growth mindset already and won’t need to be told about it.

        That’s all a separate question from the empirical evidence for that 0.1% increase. My point here is that thinking about an average effect can be misleading.

        • Andrew:

          My prior, specially in education, is that there is significant heterogeneity. So this seems plausible. Also, some papers point to effects only on under performing students which make sense to me.

          How would you go about testing your hypothesis? Both retrospectively (using the existing data from trials) and prospectively (in the design of future trials)?

    • Even with N=5000, I think that study was underpowered to detect an effect of the magnitude found in the Nature study, especially as the latter claims that it only works for poorly performing students. On the other hand, the intervention in the British study was more intensive and more outcomes were studied and there still was no inkling of an effect.

        • > IMO if you can’t detect something at n=5000, it doesn’t exist anywhere, ever.
          I continue to be surprised how widespread such beliefs are. Given that detecting an effect that is 1/10th the size requires a sample 100 times the size, it shouldn’t be surprising that there are many effects (such as as somewhat weaker version of an intervention detected with N=1000, or a version that gets lower compliance) that can’t be detected at N=5000.

          I suggest reading about large-scale, but light-touch interventions, such as advertising. There is quite good evidence of effects that are only detectable with N>100k. See such papers as
          Lewis, R. A., & Rao, J. M. (2015). The unfavorable economics of measuring the returns to advertising. The Quarterly Journal of Economics, 130(4), 1941-1973.
          Johnson, G. A., Lewis, R. A., & Nubbemeyer, E. I. (2017). Ghost ads: Improving the economics of measuring online ad effectiveness. Journal of Marketing Research, 54(6), 867-884. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2620078
          Bakshy, E., Eckles, D., Yan, R., & Rosenn, I. (2012, June). Social influence in social advertising: Evidence from field experiments. In Proceedings of the 13th ACM conference on electronic commerce (pp. 146-161). ACM. https://arxiv.org/pdf/1206.4327.pdf

          There are many other examples as well.

          See also https://twitter.com/deaneckles/status/789904354962644992

    • Quote from above: “Does this study also warrant some attention?:”

      I followed the link, and read the following conclusions concerning the study/intervention:

      “One explanation for the absence of a measurable impact on pupil attainment is the widespread knowledge of growth mindset messages: many staff members were aware of similar approaches prior to their involvement inthis project, even if they had not used them directly in their teaching. Another explanation is that Growth Mindsetapproaches take longer to embed.”

      I am not sure how to interpret the 1st explanation, but i find it worrying that there seems to be no conclusion concerning the possiblity that there simply isn’t an effect. This sounds a lot like ad hoc explanations to keep the effect and/or theory alive.

  3. But not all the studies are positive. When you get a negative one, move the bar

    https://www.tes.com/news/exclusive-growth-mindset-lessons-had-no-impact

    The EEF’s previous smaller trial of Changing Mindsets, in 36 schools, found that teachers trained in the principles of growth mindset had zero impact on the pupils they taught subsequently.

    However, when the ideas were embedded in practical workshops with pupils, children gained an extra two months’ progress compared with similar children not involved.

    While this progress was not statistically significant, the results for English were close enough to statistical significance for the EEF to conclude that the approach showed promise.

    Today’s report said that the lack of a measurable impact of the Changing Mindsets programme on pupils may be due to the widespread use of growth mindset theory – with most teachers in the comparison schools (that did not receive the intervention) familiar with it and more than a third having had training days on growth mindset.

  4. When the results are marginal, move the goalposts

    https://www.tes.com/news/exclusive-growth-mindset-lessons-had-no-impact

    The EEF’s previous smaller trial of Changing Mindsets, in 36 schools, found that teachers trained in the principles of growth mindset had zero impact on the pupils they taught subsequently.

    However, when the ideas were embedded in practical workshops with pupils, children gained an extra two months’ progress compared with similar children not involved.

    While this progress was not statistically significant, the results for English were close enough to statistical significance for the EEF to conclude that the approach showed promise.

    Today’s report said that the lack of a measurable impact of the Changing Mindsets programme on pupils may be due to the widespread use of growth mindset theory – with most teachers in the comparison schools (that did not receive the intervention) familiar with it and more than a third having had training days on growth mindset.

  5. Last year, I applied for a fellowship from the society that supported this study to use a Bayesian approach to explore the effect that regularized priors and explicit modeling of measurement error might have on inferences from these data. They weren’t interested.

  6. Some comments above are addressed/refuted by actually reading the paper. E.g. on one of the first few pages you’ll find:
    “Next, the study used analysis methods that avoided false conclusions about subgroup effects, by generating a limited number of moderation hypotheses (two), pre-registering a limited number of statistical tests and conducting a blinded Bayesian analysis that can provide rigorous confirmation of the results (Fig. 1).”

  7. Andrew said, ” I’m guessing they could do better using Bayesian multilevel analysis to partially pool estimated interactions toward zero, rather than simple data comparisons which will be noisy.”

    The last sentence of the abstract of the paper says,

    “Confidence in the conclusions of this study comes from independent data collection and processing, pre-registration of analyses, and corroboration of results by a blinded Bayesian analysis.”

    The “blinded Bayesian analysis” presumably refers to the following section of the paper:

    “Bayesian robustness analysis

    A team of statisticians, at the time blind to study hypotheses, re-analysed the dataset using a conservative Bayesian machine-learning algorithm, called Bayesian causal forest (BCF). BCF has been shown by both its creators and other leading statisticians in open head-to-head competitions to be the most effective of the state-of-the-art methods for identifying systematic sources of treatment effect heterogeneity, while avoiding false positives 40,41.

    The BCF analysis assigned a near-certain posterior probability that the population-average treatment effect (PATE) among lower-achieving students was positive and greater than zero, PPATE > 0 ≥ 0.999, providing strong evidence of positive average treatment effects. BCF also found stronger CATEs in schools with positive challenge-seeking norms, and weaker effects in the highest-achieving schools (Extended Data Fig. 3 and Supplementary Information section 8), providing strong correspondence with the primary analyses.”

    I’m not familiar with Bayesian causal forest. Can anyone who is knowledgable about it comment on whether the statement “the most effective of the state-of-the-art methods for identifying systematic sources of treatment effect heterogeneity, while avoiding false positives” is accurate?

    The references 40 and 41 are:

    40. Hahn, P. R., Murray, J. S. & Carvalho, C. Bayesian regression tree models for causal inference: regularization, confounding, and heterogeneous effects. Preprint at https://arxiv.org/abs/1706.09523 (2017).

    41. Dorie, V., Hill, J., Shalit, U., Scott, M. & Cervone, D. Automated versus do-it-yourself methods for causal inference: lessons learned from a data analysis competition. Statist. Sci. 34, 43–68 (2019).

    (I know Carlos Carvalho, but am not really familiar with his work, so can’t comment knowledgeably.)

  8. BCF is an extension of BART for causal inference that guards against the regularization-induced confounding that a simple application of BART for causal inference can sometimes suffer from. I would agree with the statement that this is most effective state-of-the-art for this problem right now. These folks (Hahn, Murray, and Carvalho) have spent years explicitly trying to solve the problem that Andrew worries about in his comment and they’ve developed an extremely effective solution.

    • Jennifer:

      I have no problem with BCF. My concern is that it was presented in the paper as a “conservative” approach, which made me concerned that it would be understating what could be found from the data. Also I’m concerned about the claim of “avoiding false positives,” as I don’t think that the concept of the false positive makes much sense here.

      In any case, let me emphasize that (a) I have a lot of respect for these researchers, and (b) there should be room for them or others to come back to these data and present a grid of inferences graphically, rather than selecting on statistical significance.

  9. I would love to see the student intervention questions. My issue with many of these studies is that the language/terminology used do not convey any substantial narrative. Maybe the references to p values are substituted for substantive explanations?

  10. If kids think their IQ (brains, smarts, capability, whatever) is fixed and doesn’t respond much to increased effort, or not enough to compensate for foregone enjoyable (possibly productive) alternative activities, then they won’t invest in those effort and sacrifices. This is basically what Prof. Dweck has been talking about all these years. That increased and better effort makes a difference in performance (outcomes) is common-sense in some places (most Asian countries) but in the USA, Prof. Dweck was needed to point it out.

    • Quote from above: “If kids think their IQ (brains, smarts, capability, whatever) is fixed and doesn’t respond much to increased effort, or not enough to compensate for foregone enjoyable (possibly productive) alternative activities, then they won’t invest in those effort and sacrifices.(…) That increased and better effort makes a difference in performance (outcomes) is common-sense in some places (most Asian countries) but in the USA, Prof. Dweck was needed to point it out.”

      I am thinking back to my youth, and i can not recall hearing much about “intelligence” in school, or at home for that matter. I DID hear a lot from my parents that “i should do my best in school”. This might be different for some kids. Aside from that, i now feel disillusioned about education, and what it’s supposed to do, and how “useful” it is. I now sometimes feel that the “do your best in school” mantra that was imprinted in my mind, may not be the most important thing i could have been taught by school and/or my parents. I sometimes felt i therefore focused on the wrong things. My point it, who knows what’s umltimately “good” for parents to teach their kids.

      Regardless of parental guidance, and influence, in this all: i think it was, somehow, very clear at school that you should do your best and learn the things they taught. I find it hard to believe how kids in school do NOT somehow (repeatedly) get that message. So in regard to that, i don’t understand why an intervention like “growth mindset” is necessary when it comes to things like effort, and doing your best. I am also concerned about outside interventions in general in schools, as i reason they can easily be abused when it comes to money, and other things.

      • When someone decides whether they’ve done their best, they are to some extent constrained by their assessment of their own abilities. When a problem gets difficult, a child has to decide whether it’s potentially solvable or if they simply lack the ability to do so, no matter the effort. My understanding is the growth mindset is supposed to promote the belief that continued effort will pay off and that one should not rely on self-assessed ability to decide whether to keep pushing through difficult problems (since the underlying ability may change in response to the effort).

        • Quote from above: “My understanding is the growth mindset is supposed to promote the belief that continued effort will pay off and that one should not rely on self-assessed ability to decide whether to keep pushing through difficult problems (since the underlying ability may change in response to the effort).”

          I think this is potentially a “wrong” thing to communicate, especially if it’s not how things truly work. I think this is also potentially not very smart. I also think this sends out a wrong message (e.g. my life sucks, so i must not have worked hard enough). And i think this teaches kids, and adults, to be some sort of “hard worker” and not so much a “person”.

          (Luckily?) i don’t think that is exactly what they communicated to the kids during the intervention. I browsed through the few screenshots i could find in my little search of the specific “growth mindset” study. In it i could find a screenshot that seemed to be about “It’s not just about the effort, but also the strategy”. That made me glad to see that. Glad concerning not mindlessly “grinding it out”, and glad concerning teaching and/or giving the kids themselves the choice and ability to stop and try something else.

          So, for me it’s still not clear what “growth mindset” actually specifically means or implies. Especially in relation to things like (the possibility of) inherent differences between people in things like intelligence, capabilities, etc., and in relation to (the possiblity of) more changeable things like motivation, effort, etc.

          I also wonder why the “growth mindset” seems to only be about grades in school and not getting “good” at sports, or playing a music instrument, or drawing, or building stuff with legos, or taking care of pets, or cooking, or making friends, or skateboarding, or programming, or doing ballet, or rapping, or dancing, or swimming, or hunting, or fishing, or making art, etc. (it should be clear by now by the several comments in this thread that i kind of dislike all this school and “education” stuff).

    • From the wikipedia page:

      “The story is used to teach children the value of optimism and hard work. Based on a 2007 online poll, the National Education Association named the book one of its “Teachers’ Top 100 Books for Children”.

      Again, i am not a big fan of the school system and “education” so i think i’ll leave it at that…

    • I was somewhat familiar with “the little engine that could” but never knew of the exact story. I looked it up, and found a possibly interesting interpretation somewhere which went like this:

      “The train breaks down.
      The little clown finds the passenger engine.
      The passenger engine rejects them, and that’s a bummer.
      The little clown finds the freight engine.
      The freight engine rejects them, and the dolls and toys feel pretty disheartened.
      The little clown finds the old rusty engine.
      The old rusty engine rejects them, and the dolls and toys are ready to have a good ugly cry.
      The little clown finds the Little Blue Engine.
      The Little Blue Engine is willing to give it a shot, and you know the rest of the story.

      Here’s the thing – if we really want a story about perseverance, grit and faith, it is not the story of the Little Blue Engine. It is the story of the little clown.”

      It’s the freaking little CLOWN that could!

      • Quote from above: “It’s the freaking little CLOWN that could!”

        I did a little more searching and reading. If i understood things correctly, there are several versions of “the little engine that could” story. Not every story has a clown in it for instance.

        I tried to find the most early version of the story, and in the Wikipedia link provided above i could find a link to a 1906 “Story of the engine that could”. I subsequently read it, and wondered about the following:

        In that 1906 version it was the superintendent asking all the engines for help. So that’s actually very similar to the clown asking for help, so in that regard the point that the story could actually be interpreted as not being about the train that could, but about the superintendent or clown that could may actually still hold!

        Regardless of that, i read the following:

        “The superintendent of the yard was 
not sure what it was best for him to do, so he
 went up to a large, strong engine and asked:
 “Can you pull that train over the hill?”

        “It is a very heavy train,” responded the engine.

        He then went to another great engine and 
asked:
 “Can you pull that train over the hill?”

        “It is a very heavy grade,” it replied.

        The superintendent was much puzzled, but he 
turned to still another engine that was spick
 and span new, and he asked it:
 “Can you pull that train over the hill?”

        “I think I can,” responded the engine.
”

        Now, what i immediately thought was that “the heavy train”, and the other “great engine” did NOT nevessarily say that they wouldn’t, or couldn’t, want to try and move the heavy train over the hill. In my view, they just stated a fact: that the train was heavy, and that there was a heavy grade.

        In my interepretation, these remarks could just be the start of thinking about whether it was possible, or smart, to move the train up the hill. The superintendent does not know what to do according to the story. It seems to me that the first 2 trains he asks are just about to start talking and thinking about the question to help the superintendent, but this superintendent seems to have no patience! The superintendent does not seems to have time for all this thinking, and quickly moves on to the next train to ask the question.

        When the superintendent finally arrives at the new engine, it said “it could”, and that’s the end of it. There is no deliberation concerning whether it is a “good” idea to let the new (inexperienced?) engine attempt this all. There is no reflection on the assessment of situation, that is backed up by the 2 other trains, that the train, and the grade, might be very heavy.

        So, an alternative interpretaion of this story could also be that the superintendent ask for help because he does not know what to do, but is then subsequently inpatient, and does not seem to think about the situation much. He then, without much further pondering, accepts that a “spick and span new” (inexperienced?) goes out and attempts to pull the heavy train at a heavy grade over the hill.

        I think the superintendent might have been irresposible, and did not thinkt things through. Perhaps this new engine got lucky in succesfully getting over the hill. Perhaps there are dozens of other new engines that didn’t make it and crashed the train. And perhaps there are dozens of other engines that will crash if attempting the same thing.

        If the superintendent had been more patient, thought about things more, and listed to the 2 engines, they could have come up with a better solution…

        • Quote from above: “Perhaps this new engine got lucky in succesfully getting over the hill.”

          But at what potential cost? And have we heard since from this “spick and span new engine”?

          Perhaps our little hero was so exhausted after getting over the hill that it was never the same after that anymore. Exhausted from it’s efforts, it took a break from things. It then, to pass the time, did some betting on greyhound-racing. Sure, it was fun at first. Especially because he had a few nice winnings, but after a while things went wrong.

          When the engine was physically capable of getting back to work, it wasn’t mentally anymore because of the gambling addiction and (newly developed) substance abuse. The superintendent fired the “spick and span new eninge” because he just didn’t have the patience to wait for the new engine to become well again. He hired a “brand new spick and span” engine to take the place of the hero of our story.

          As a result of that, the once “spick and span new engine” became less spick and span. It became dirty, and the wears and tears showed more and more. Other wear and tear did not show on the outside as much, but it was still there. And this wear adn tear that didn’t show on the outside had an effect. “The engine that could” subsequenlty lost his home, because it had no job anymore, and because of the now cripling debt due to the greyhound-racing stuff.

          Now being homeless, the (once) “spick and span” new engine wandered aimlessly across the land. Then all of a sudden, it passed by the exact railroad that he went up the hill on a long time ago. What he didn’t spot at the time, he spotted now. He saw lots of other engines besides the track laying still covered by the shrubs, and bushes. If you looked closely, you could see some where hanging from the trees. They all didn’t seem to move, looked terrible, looked rusty, and some of them had giant holes in them.

          “The engine that could” wondered if it was “the engine that shouldn’t have”.
          “The engine that could” wondered if it should have been “the engine that didn’t”.

  11. I’m all for the growth mindset too, not just for an hour, but every day.

    Just the same I’m sure this study won’t hold up. Even as small as the claimed benefit is, it can’t possibly result from a one-hour intervention. People go to therapy for years to generate self confidence. The idea that it can be done in a huge population of children in one hour just doesn’t hold any water.

    The old adage holds: extraordinary claims require extraordinary evidence.

    • The thing you have to watch out for here is this:

      The fact that they’ve checked all the replicability boxes doesn’t make the conclusions a revelation of truth. All that does is bring it up to the ordinary standard. Now it’s actually worth while to evaluate and critique the methodological details, results, analysis, and conclusions. Whether it survives that critique is anyone’s guess.

      • Quote from above: “The fact that they’ve checked all the replicability boxes doesn’t make the conclusions a revelation of truth. All that does is bring it up to the ordinary standard. Now it’s actually worth while to evaluate and critique the methodological details, results, analysis, and conclusions. Whether it survives that critique is anyone’s guess.”

        Yes!

        A point (people on) this blog has often made (“Transparency and openness are not enough”).

        I tried to make (the gist of) this clear in a recent different discussion as well:

        https://statmodeling.stat.columbia.edu/2019/07/18/pre-results-review-some-results/#comment-1092825

      • Quote from above: “The fact that they’ve checked all the replicability boxes doesn’t make the conclusions a revelation of truth. All that does is bring it up to the ordinary standard. Now it’s actually worth while to evaluate and critique the methodological details, results, analysis, and conclusions. Whether it survives that critique is anyone’s guess.”

        I have began to wonder where this sometimes seemingly singular focus on “transparancy” and “openness” may have come from. I think it all started in 2012 with the paper by Nosek, Spies, & Motyl “scientific Utopia II: Restructuring incentives and practices to promote truth over publihsability”.

        In it there is a chapter/headline/whatever the correct term is, that is called: “The ultimate solution: Opening data, materials, and workflow”. I think it’s almost always “wrong” to write things in scientific papers using this kind of language, but leaving that aside i think “opening data, materials, and workflow” may have little to do with “solving” the actual problematic issues.

        Regardless, i have wondered if that’s where it all started. Of course, this message of “transparency” and “openness” has since been repeated over and over and over again. If i am not mistaken 2 of the authors of that paper (Nosek & Spies) subsequently became “directors” of some sort of institute that’s supposedly all about promoting “transparency”. I am not sure how to view that.

        I have since wondered how much money a “director” of such in institution makes. Have people like Nosek & Spies have ever been “open” and “transparent” about their salaries there or where all this money is exactly going to in their institute, while they are asking lots of people to help them out in their “collaborative” efforts?

  12. Anon said,

    “If i am not mistaken 2 of the authors of that paper (Nosek & Spies) subsequently became “directors” of some sort of institute that’s supposedly all about promoting “transparency”. I am not sure how to view that.

    I have since wondered how much money a “director” of such in institution makes. Have people like Nosek & Spies have ever been “open” and “transparent” about their salaries there or where all this money is exactly going to in their institute, while they are asking lots of people to help them out in their “collaborative” efforts?”

    I believe you are thinking of the Center for Open Science. Their website is at https://cos.io .

    Also see https://www.charitynavigator.org/index.cfm?bay=search.profile&ein=461496217 for Charity Navigator/Guidestar information about the Center for Open Science.

    • Quote from above: “Also see https: //www.charitynavigator.org/index.cfm?bay=search.profile&ein=461496217 for Charity Navigator/Guidestar information about the Center for Open Science.”

      It seems like i have to log in or register to view the “IRS forms 990”, which i think gives me the kind of information i was wondering about (i.c. salaries, where the incoming money is being spent on exactly, etc.). I don’t want to register, but perhaps the “Center for Open Science” can provide these type of reports themselves. I would like to know such thing, for instance before i decide to help them out with one of their proposed “collaborative” project. I think that also would fit nicely with “openness” and “transparency”.

      On a more general note. I am wondering more and more 1) if it is “fair” to ask (young) people for help in “collaborative” efforts while you possibly reap all the benefits, and 2) how much all these “directors” of big projects that are supposed to improve science like Nosek and Spies earn. It already seems to me that coming up with a new “let’s improve science” project is similar to the “sexy study” of a decade ago: both can lead to asking lots of money, of which it is often not clear where it exactly goes.

      I find the case of the “Center for Open Sciene” specifically interesting (and somewhat worrying) in a way. I think i was around when it got established, so i sort of have been following them for a while. I find this al interesting because they seem to be viewed as some kind of “scientific heroes” and “super scientists” that will fix everything by some, and they seem to me to receive lots of financing and help from other scientists.

      I on the other hand am wondering more and more what they effectively do and contribute and if that’s worth all the money they have spent. I also sincerely question their capabilities and methods, and/or whether their goals are the “right” ones concerning actually improving science. And i find it strange, and wonder about the possible ethical part of, that they ask for lots of (young) people to help them out, while they themselves are probably the ones benefiting the most from it all. They seem to be getting more and more media attention, power, influence, and money.

      To me, they seem to do lots of things that 1) either already exist or have existed, and/or 2) ask and/or receive and/or give a lot of power and influence and benefits to a small group of people, and/or 3) mess things up and/or make things even worse.

      Please correct me if i am wrong about any of the following, and please note that i may make mistakes! I will provide information where i can, but don’t want to provide many links because that often results in my comment not being depicted on this blog. I think the following provides enough information though for people to verify things for themselves should they want to. Here we go:

      Concerning “1) they seem to do lots of things that 1) either already exist or have existed”:

      # The “Open Science Framework” is nothing new really in my view. It just combines a lot of things already in existence, like providing the opportunity for pre-regsitration, and data storage. It has a lot of “fancy” functions perhaps, like “forking” things but i doubt this is used much, and/or very useful.

      # They promoted, and/or (say they) designed, “open practices badges” and “Registered Reports” which may have both been proposed and/or executed before. “Open practices badges” look a lot like the 2009 “Biostatistics” introduction of “kitemarks”, and “Registered Reports” look a lot like a 1997 “Lancet” publication pathway, and an idea proposed by M. J. Mahoney in 1976 in A book called “Science as subject”.

      Concerning “2) ask and/or receive and/or give a lot of power and influence and benefits to a small group of people”:

      # They seem to promote the h#ck out of “Registered Reports” which in my view gives more and more unnecassary influence to editors, journals, and reviewers by a) promoting, and praising, the role of the reviewer in stage 1 of “Registered Reports”, and b) making “pre-registration” something “special” between the author and the journal/editor/reviewer system, instead of something that is simply between the author and the reader.

      If i understood things correctly, there was/is also talk of setting up “special editors” to handle “Registered Reports”. In my view this is giving the exact people and system that helped mess things up unnecessary power and influence.

      # They seem to promote the h#ck out of “Open Practices Badges” which in effect may give additional “special” power and influence to journals and editors IF they are the only ones who can “hand out” the badges. If i am not mistaken there was/is also talk of possibly given a “3rd party” the power and influence to hand out badges. In my view this would again be giving the exact people and system that helped mess things up unnecessary power and influence.

      # A lot of their efforts and promotions seem to all be big “collaborative” projects, which are currently being called “crowdsourced” projects by some, if i understood things correctly. These “big projects” in my view ultimately give power, and influence, and benefits to a small group of people.

      To me, these “big projects” (will) probably involve all the things that i think have contributed to the current mess in academia/science: money, power, influence, politics, etc. To me it’s like they mistook the critical paper by Binswanger (2014) “Excellence by nonsense: The competition for publication in modern science” for a playbook!?

      Concerning 3) mess things up and/or make things even worse.

      # “Registered Reports” seem to have been designed and/or implemented so poorly that there often isn’t even any “registration” and/or the actual reader of the papers are not able to view the registration (e.g. see Harwicke & Ioannidis (2018) “Mapping the universe of Registered Reports.

      # The “Open Practices Badges” paper, and study, that “evaluated” the badges may have been written and/or designed poorly (e.g. see https: //blogs.plos.org/absolutely-maybe/2017/08/29/bias-in-open-science-advocacy-the-case-of-article-badges-for-data-sharing/).

      And the actual implementation at the journal “Psychological Science” of the “Pre-registration badge” seems to not have let the reviewers check things very carefully resulting in papers that did not accurately describe things that were written in the pre-registration. See “Preregistration: Comparing dream to reality” by Claesen, Gomes, Tuerlinckx, & Vanpaemel (2019). What are these editors and reviewers even doing if they apparently can’t even proparly check things like the pre-registration?!

      # Another one of their efforts, the “TOP-guidelines”, seem to have incorporated both “Registered Reports” and “Open Practices Badges” in the guidelines, without any solid evidence that these things are even “good” for (improving) science. Given the possible problematic issues presented above concerning these projects, i view this as even more irresponsible.

      So overall, i am sincerely wondering if they are helping to improve Psychological Science or making it even worse. And i am wondering how much money people who do, or may have done, things like the things described above make. I honestly have no idea. Like would “directors” like Nosek make more than 100000 dollar a year? Could they make 500000? I honestly don’t have a clue.

  13. It seems that the study was carefully designed and implemented. It also seems that the researchers have moved away from a theory of two opposed minsets, growth and fixed, and are focusing instead on helping students see how they *can* grow, not only in knowledge, but in ability.

    I have two qualms. One is over the use of either GPA or course enrollment as an indicator of intellectual growth (or even belief in the possibility of growth). Grades in the US–in mathematics and sciences, as well as in other subjects–are awarded for all kinds of things–homework, tests, class participation, group projects, extra credit work–and the criteria can vary widely from one classroom to another. I am not convinced that a tenth of a point’s difference in math and science GPA means anything at all. One would have to look more closely at the grades’ basis to see what this difference meant.

    Enrollment in advanced mathematics classes, too, seems a hazy and dubious indicator of growth or belief in growth. Yes, it is good to take on challenges, but signing up for an advanced course is only a preliminary step. The student must then do the work and learn the material. It would be interesting if the researchers were to follow up and see how the students did in these courses.

    My second qualm, which I have brought up in my book and elsewhere, has to do with the very concept of growth mindset. I have argued many times that we not only have a mixture of mindset but actually benefit from the mixture–that we need a sense of limitation as well as of possibility. It is fine to know that one is better at certain things than at others. This allows for focus. Yes, it’s important to know that one can improve in areas of weakness. And one’s talents also contain weaknesses, so it’s helpful, overall, to know how to improve and to believe that it can happen. But it does not have to be an all-encompassing ideology, nor does it have to replace all belief in fixity or limitation. One day, someone will write a “revelatory” book about how the great geniuses actually knew they were bad at certain things–and how this knowledge allowed them to focus. That will then turn into some “big idea” and go to extremes of its own.

    • Diana:

      I agree, especially on your last point. Just speaking qualitatively, as a student, teacher, sibling, and parent, I’d say the following:

      – When I first heard about growth mindset as an idea, 20 or 30 years ago, it was a bit of a revelation to me: one of these ideas that is obvious and that we knew all along (yes, you can progress more if you don’t think of your abilities as fixed) but where hearing the idea stated in this way could change how we think.

      – It seems clear that growth mindset can help some kids, but not all or even most, as these have to be kids who (a) haven’t already internalized growth mindset, and (b) are open and receptive to the idea. This is an issue in learning and persuasion and change more generally: For anything, the only people who will change are those who have not already changed and are willing to change. Hence a key to any intervention is to target the right people.

      – If growth mindset becomes a dominant ideology, then it could be that fixed-mindset interventions could be helpful to some students. Indeed, maybe this is already the case.

      The interesting thing is how much these above principles would seem to apply to so many psychological and social interventions. But when we talk about causal inference, we typically focus on the average treatment effect, and we often simply regression models in which the treatment effect is constant.

      This suggests, in a God-is-in-every-leaf-of-every-tree way, that we’ve been thinking about everything all wrong for all these decades, focusing on causal identification and estimating “the treatment effect” rather than on these issues of receptivity to treatment.

      • “This suggests, …, that we’ve been thinking about everything all wrong for all these decades, focusing on causal identification and estimating “the treatment effect” rather than on these issues of receptivity to treatment.”

        Interesting idea. And it brings up the question: Who is receptive to *which* treatment? — i.e., what produces desired changes in some people might not produce desired changes in other people. while what produces desired changes in the second group does not produce desired changes for the first group.

  14. If one were interested in in using a ML BCF model, how would one practically implement this in R? While the authors describe their approach in the supplemental section as well as in various talks, I am still unsure as to how one would fit a ML BCF model.

Leave a Reply to Shane Tutwiler Cancel reply

Your email address will not be published. Required fields are marked *