Yuling writes:
I would like to point to you and your readers two papers that address the same scientific question but come with opposite conclusions:
Both https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2786138 and https://www.frontiersin.org/articles/10.3389/fpubh.2021.622379/full assess mortality in rural Bangladesh in relation to Covid-19 using large scale survey data. But one paper concludes that “all-cause mortality in the surveyed are was lower in 2020 compared with 2019” and the other says “The study reported a 28% increase in excess deaths among the elderly population during the first months of the pandemic.”
These are different studies and they come to different conclusions. Just reminds us of the everyday difficulties of doing science.
P.S. Here’s a fun page of contradiction quotes.
It’s just because looking at a dataset in different ways can tell us conducting things.
I am sure that J meant to say
“It’s just because looking at a dataset in different ways can tell us ‘confusing’ things.”
rather than
“It’s just because looking at a dataset in different ways can tell us conducting things.”
Or, perhaps the original was meant and “conducting” has a technical meaning I am not aware of.
Contradicting, I guess.
Yep, meant contradictory. Statistical analysis is frequently not conclusive. I’d say that’s a significant statement at about p<0.05, but your analysis of the data might lead to a different conclusion!
I have often thought about attempting to take a random introduction of some social science paper, and re-write the sentences to make them state just about the opposite of the original sentences. I would then try and find papers to support these new sentences and conclusions.
I though it would possibly point to several problematic issues in social science and publishing and all that stuff. I can’t be bothered anymore, but I did something similar on this blog regarding some social science study that was mentioned on here which sort of quenched that thirst.
It would be fun to disentangle this, but I don’t have the energy. Looking at the 2 studies I see (at least) the following issues: different study areas (same country but not the same geographical areas), different data collection (surveys vs observational), different methodologies (too many researcher degrees of freedom to list them all), and too many contradictory points with less-than-clear ways to compare them. I did notice that the raw mortality rates appear to differ greatly between the two studies – for the elderly, it looks like one study reports mortality rates around twice as high as the other. If I am reading that correctly, then I’d focus on either the different study populations or conflicts between the survey data and the observational data, perhaps not needing to delve too much into the estimation methodologies. Maybe someone with more time on their hands, energy, or just a quicker study than me, can explore this more. It could make a good exercise for a class.
I’m not sure why different results from two different places would be considered contradictory, unless you held the belief that all of “rural Bangladesh” was infused with some essential similarity that made different results a puzzle. If two studies asked the general scientific question, “How has the number of species of breeding birds in terrestrial communities in Great Britain changed in the last 20 years?”, a study in Scotland might well give a different answer than a study in Wales.
Not having read the actual papers, I don’t see a contradiction in the two statements. The first is (implicitly) about the whole population for the whole year, while the second is about a subpopulation for some number of months.
Yeah. I read it repeatedly to see what was contradictory but not finding it. I was thinking I must be missing something (happens a lot with me).
So – I was glad to see your comment.
What do you think of the “all-cause mortality in the surveyed are[a] was lower in 2020 compared with 2019” conclusion?
It seems to be based on the “after adjustment for survey nonresponse and poststratification, 2020 mortality changed by −8% (95% CI, −21% to 7%)” result.
Since “lower” masks the fact that the change was NOT significant, the conclusion sounds rather misleading.
Figure 2 in the first article is quite confusing.
It’s presented as “The age distribution of the surveyed population and associated mortality rates in our data were comparable to published national trends (Figure 2).”
The title of the first two panels is “Deceased population (2019/2020)” and the y-axis is labeled “Proportion of population, %”.
Actually these charts show densities (integrating to 1 over the range from 0 to 85).
They have nothing to do with “mortality rates” and while the have the shape of proportions the labels are misleading (the proportion of deaths in the 80+ bucket is over 25% for 2019).
The 2019 mortality rate for the 80+ group is above 14% which is “comparable to published national trends” in the sense discussed recently here (any two things can be compared) but it is substantially higher than the figure for the region (12%) and well above the national average (both rural and urban are around 11%).
Maybe there is something about 2019 – or an issue with the data – that could explain this large mortality and the decline in mortality in the group aged 80 years or older driving the “lower mortality” result could be a regression to the mean from an abnormally high baseline.
They may also be undercounting 2020 deaths – given that 25% of the households dropped out from the survey. “Our approach to calculate aggregate mortality using demographic group-level mortality corrects for this, but only to the extent that mortality estimated for a particular age, sex, and education group is not biased by nonresponse. […] We also cannot exclude that households with a member who recently died may be less likely to pick up the telephone or less willing to participate in a survey.”
(Actually they may also be undercounting 2019 deaths slightly by asking 2020 households about the deaths in the household in the previous year. People living alone in 2019 are either alive in 2020 and reporting no deaths or dead and nobody asks them to report their own death.)
A household where someone died has one fewer person who might be around to answer the phone, for one thing!
And how will AI untangle all of this? Will we have Bayesian AI and frequentist AI slugging it out on social media?😎