Michael Stutzer writes:
This study documents substantial variability in different researchers’ results when they use the same financial data set and are supposed to test the same hypotheses. More generally, I think the prospect for reproducibility in finance is worse than in some areas, because there is a publication bias in favor of a paper that uses a unique dataset provided by a firm. Because this is proprietary data, the firm often makes the researcher promise not to share the data with anybody, including the paper’s referees.
Read the leading journals’ statements carefully and you find that they don’t strictly require sharing.
Here is the statement for authors made by the Journal of Financial Econometrics: “Where ethically and legally feasible, JFEC strongly encourages authors to make all data and software code on which the conclusions of the paper rely available to readers. We suggest that data be presented in the main manuscript or additional supporting files, or deposited in a public repository whenever possible.”
In other words, an author wouldn’t have to share a so-called proprietary data set as defined above, even with the papers’ referees. What is worse, the leading journals not only accept these restrictions, but seem to favor such work over what is viewed as more garden-variety work that employs universally available datasets.
Intersting. I think it’s just as bad in medical or public health research, but there the concern is sharing confidential information. Even in settings where it’s hard to imagine that the confidentiality would matter.
As I’ve said in other such settings, the authors of research papers have no obligation to share their data and code, and I have no obligation to believe anything they write.
That is, my preferred solution is not to nag people for their data, it’s just to move on. That said, this strategy works fine for silly examples such as fat arms and voting, or the effects of unionization on stock prices, but you can’t really follow it for research that is directly relevant to policy.
One of my favorite not dead horses. I would add one thing to Andrew’s take (which I agree with): when the research can impact public policy makers, they should insist that the data be provided either publicly or at least to the journal’s referees. Plenty of academic research is aimed at influencing policy, either directly or indirectly. As long as the decision-makers (politicians, regulators) accept the results as credible (at least somewhat), they are complicit in the potential abuses (whether they be intentional or innocent accidents). I may have no obligation to believe what they write, but I’d like to see the decision makers emphasize that point. When they begin disbelieving what is written, then we might see a movement towards release of this data. As long as they remain silent, they are implicitly condoning such practice.
While I agree there is a problem, please don’t give decision makers (politicians) another reason to ignore scientific analysis — the current political climate is already tending anti-intellectual, and further encouragement is likely to do more harm than good.
The cryptographic community has developed multi party computation and homomorphic encryption which allows statistical calculations such as regression to be performed without revealing the raw data. Of course there’s a practical price that one must pay, but things are getting much more efficient. https://en.wikipedia.org/wiki/Secure_multi-party_computation?wprov=sfti1
This is fascinating – but I’ll admit I understand (at best) 5% of that link. Can you (or someone else) explain in common language (I guess I could try ChatGPT) how these techniques could be used in the case of private data. What regression models can be run and how would they reveal the accuracy of the underlying data? And how does this address the non-standard errors referred to in the post (the garden of forking paths issues)?
In reality I think this just lets you prove the following kinds of theorems:
running a linear least squares regression with ABC specification on this secret dataset produces the following coefficients…
Basically it doesn’t address any of the real problems which are things like validity of the data itself, measurement errors, graphical displays of residuals, alternative specifications, etc.
It’s been a while since I’ve had to work with MPC but I can maybe try to give some context and try to point to additiona resources. As it says in the link a set of partcipants want to compute a function F(d1, d2, …, dN) without revealing their data – the point being the only information gained by the other parties at the end is whatever is revealed by the output of F itself (“The only information that can be inferred about the private data is whatever could be inferred from seeing the output of the function alone”). A traditional example is computing the AND of the bits of all participants (essentially to see if everyone agrees on some proposition but without revealing the one or several people who disagree). A more practical example is probably Google’s password breach detection https://cloud.google.com/recaptcha-enterprise/docs/check-passwords
The more concrete question is just how this could be used practically. On one hand there are very strong results regarding what types of functions can be realized as MPCs.There are different settings where MPC can be defined which I don’t feel comfortable going into without actually rereading the relevant papers, but I’ll paraphrase Lindell (https://eprint.iacr.org/2020/300.pdf) in that it can be shown that given some essentially realistic cryptographic assumptions and settings any function can technically be securely computed (for example https://eprint.iacr.org/2002/140), although this does not consider the practical costs incurred (there have been some surveys on modern compilers such as https://ieeexplore.ieee.org/document/8835312 but again this is outside what I’m comfortable giving any detailed exposition on).
I don’t have enough knowledge on the current state of implementations of MPC to give concrete examples of what is genuinely possible, but this github: https://github.com/rdragos/awesome-mpc has at least some links to implementations of concrete ML-libraries using MPC. We can definitely view plots as functions so private computation isn’t just limited to computing sums (here’s a pretty plain English look at private histograms: https://jamey.thesharps.us/2018/07/10/private-secure-multiparty-histograms/).
Like Daniel says I don’t think private computation protocols are inherently some silver bullet regarding the non-standard errors, since ultimately they are just a way of implementing (any) function in a way that doesn’t reveal the underlying data to other participants. Technically this means that anything that can be put in terms of a function or an ensemble of functions could be (at the very least theoretically) realized via MPC, but I don’t think there are any out-of-the-box solutions for this type of problem yet.
I think the privacy problem with medical research data is overblown. While there are some situations where disclosure could lead to serious harm, and those really should be protected, they are uncommon. Most data used in medical research could be published, even without removing identifier variables, and nobody would be any the worse off for it. This is especially true in the contemporary environment where most of the agents that would seek to gain advantage from the information probably have other ways of getting it anyway. (E.g., notwithstanding the ostensible legal protection of my health information, I have no doubt that anybody who might gain from knowing my complete medical history has already figured it out by purchasing my web browsing history or other readily available data and metadata that normal life generates.)
A less tractable problem arises with technology assessment–which is very policy relevant. New medical technologies are developed in a highly competitive, and highly secretive environment. These businesses simply will not release data to researchers without ironclad non-disclosure agreements, for obvious reasons. The researcher’s only options are to keep the data strictly secret or forego the opportunity to do the research. I think this raises important ethical issues, and I don’t see any good solutions to this problem.
In my own work in this area, I make it clear in my publications that the data was provided confidentially by the company that developed/manufactured/marketed the technology, as the case may be. At least this puts the reader on notice that the data, and results derived from it, cannot be independently verified, so make of it what you will. But this really isn’t satisfactory.
I completely agree with your first paragraph. HIPAA has provided an easy excuse for most medical data used in research to be withheld. Perhaps it was with good intentions, but it has proven to be a hidden cost of that legislation (similarly, in the case of educational data due to FERPA). Ironically, the best data is often at the hands of insurers, where their private data can be shared with each other. The most accessible data I have found in the US is Medicare data, but not all medical questions involve the Medicare population. Much of the available public data comes from the NHS, although it is much more accessible to UK researchers than US researchers.
In our litigious world, if you can imagine a misuse for data, then it is grounds for walling off access to it. And, unfortunately, it is ever easier to imagine misuse. The open data movement has an uphill battle.
I also agree with your second paragraph and don’t see any easy answers either.
You describe HIPAA as an “excuse”. It’s a law. You can be fired, fined, and go to jail. I’ve seen it happen in the hospital where my laboratory is located.
Maybe I misinterpreted your meaning.
It is an “excuse” in a more global meaning. I’m not saying that individuals have a choice, but as a society we have a choice – and the choice we made serves as an “excuse” for not sharing anything.
I think a lot of the purported solutions for privacy are worse than the problem they try to solve.
Take the EU GDPR. Granted, it may have good parts, but to the extent that it makes me click “accept cookies” on different website popups a dozen times a day I can’t believe it has, on net, made the world any better?!
Check out some GDPR block lists for uBlock. Not perfect but pretty good.
Clyde would you participate in medical research if you knew your identifying data would be made publicly available? Do you think that other people think like you? Do you think people would participate in sexual health research, food diaries, irritable bowel syndrome studies, studies about mental illness or suicidal ideation, or any of a range of icky and uncomfortable or socially stigmatized conditions if they knew their names would be publicly available? Medical research would grind to a halt for lack of data if we shared identifying variables.
I won’t pretend to represent Clyde’s views, but I share his belief that privacy concerns are overblown. Plenty of medical data is not sensitive – I really don’t care if my entire medical history is made available publicly. But you cite a number of areas where the concerns are real and damage to people is entirely possible. Making data publicly available does not mean publishing the identify of people. De-identification is entirely possible. It is true that very competent people can break many or most de-identification schemes, so care must be taken. All I would ask for is some sense of balance between the potential harms and the degree to which efforts are made to protect identities. Not everything in a data set needs to be protected.
What we currently have (for at least medical and educational data) is an extreme position where everything is protected – and this means that the published analysis is “protected” against attempts to validate the research.
+1
Over the course of my life, in addition to doing medical research, I have been a subject in 6 studies. All of those were before HIPAA was enacted. None of those involved sexual health, food diaries, or irritable bowel syndrome, but one was a mental health study. None of the other five I participated in involved anything sensitive.
No, I would not have participated in the mental health study if I expected that individual data would be made public. For the other five studies, however, it would nothave mattered to me at all. And even for the mental health study, I would have been perfectly happy for identified data to be shared with other researchers for the purposes of replicating or verifying aspects of the study.
While I have never done a formal survey of the matter, I do think that most people think about this more or less as I do. In the course of recruiting study participants for studies that do not involve sensitive information, I cannot recall ever being questioned about the security of the data (including before HIPAA).
There is a distinction between sensitive information and non-sensitive information that has been lost in the current environment. And it has been my experience that a large majority of medical research uses only non-sensitive information.
And thus you highlight the issue here, Clyde. Imagine that other people have different triggers and sensitive spots to you, and you can see that there will be a lot of studies where people will refuse to participate if they know their data is being shared. It’s also unlikely that this refusal is random, it is likely related to presence/absence of symptoms, or their severity, and will naturally introduce bias. This is likely the case even if direct identifying details aren’t going to be released, since many people have different thresholds for what they consider to be the risk of identifiability. Consider for example the diversity of parental views on putting children on social media, from using them to make content down to refusing to let any pictures of them appear ever. People have vastly differing views of what the risks of public disclosure are, or what constitutes public disclosure.
Also, imagine for a moment that you aren’t a comfortable male academic, but are (for example) a woman fleeing an abusive relationship, a person who recently moved from NY to Florida to take up a high school teaching position, a person with tattoos who recently moved to Japan but did a study on hepatitis C risk in rural Australia, someone who had an abortion back when it was legal in the USA and participated in a study about it … the list of people whose lives can be turned upside down by the disclosure of potentially identifiable data is huge. Just to take the tattoo example, it would be easy to identify someone from rural Australia in a study even if it didn’t include names, and having a tattoo can get you fired in Japan. A malicious colleague doing a search might find such a thing.
Bear in mind that various US agencies won’t release aggregate data if it includes counts <10 (or <6, or 5, depending on the agency) just in case someone could be identified. This includes death data from years back, even though the person who could be identified is dead and presumably doesn't care. I think we all agree that this kind of security is overwrought, but we should also consider the possibility that these rules are in place as a consequence of bitter experience.
I took a look at the paper. I saw some markings and comments in there- were they intentionally left in there for us blog readers to find and ponder?
Raphael:
I have no idea. That’s just the version that was sent to me.
If you look at the famous paper by Simonsohn et al. (2011) about data analysis flexibility and what the results of that might be for research findings, I would reason that the importance or usefulness of the availability of data in and of itself might be severly overstated.
Now if it’s accompagnied with pre-registration information, I am willing to take it more seriously.
Quality peer review processes where the data, pre-registration info, code provided, etc. necessitates so much work for the peer reviewer, that overhauling the peer review system seems like a prerequisite for the rest… but none of us want more unpaid work, so… what can we do?
What can we do?
I would say nothing.
Simple might be better.
Less might be more.
I reason that the entire publishing system with peer-review and gate-keeping has probably done more harm than good. I also reason that many (future) proposals to change things probably lead to no real change, and/or are merely different forms of the same problematic issues, and/or may cause a different set of problems.
I wrote some thoughts about how I view “publishing” and “submitting”, where I focused on the definitions of these words in relation to the currently common publishing system. I thought it was amusing and/or noteworthy to do this, and it also made it clearer to me why I probably don’t like how things are currently going.
I wanted to quote some things here from that manuscript, but found it too hard to do in a few quotes. The general point I make is that laregely everything should just be pusblished with no “peer-review” in the traditional sense. If, and how, papers are subsequently used is largely all the peer-review and quality control that is perhaps needed and desirable.
The manuscript is titled “Things I have wondered (so far)” and can be found on SSRN. The short section on submitting papers, and publishing, etc. starts on page 7.
I don’t think they really care if you (or I) believe them. As long as they keep getting funding they will continue doing whatever it is. Be it not sharing data, testing strawmen hypotheses, p-hacking, or whatever.
It is the standards of the funding sources that matters.
This is exactly right
Ben Goldacre has shown how this can work with sensitive NHS data. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1067053/goldacre-review-using-health-data-for-research-and-analysis.pdf
There is only one copy of the raw data, that is never shared, but users can look at the variables and request an analysis.
He’s got the platform working, so it presumably could be rolled out in other contexts
Brilliant!
I asked for one correlation coefficient from data on ~4000 physics graduate students, needed to help evaluate a very influential (and completely wrong) paper on whether GRE’s were predictive. I was told it was unavailable because of “human subjects”.
At least they were willing to admit the subjects were human. I wonder if the IRB feels that that reveals too much about them,
There are a lot of problems in practice with this absolute transparency approach, and I don’t think anyone who advocates it has a good solution or answer to them. I wonder if you have any thoughts? Here are some examples of the problems.
1) I do research with sexual minorities, including young people, concerning stigmatized and sometimes illegal behavior. I don’t think that a single person would agree to respond to my surveys if I wrote “Your data will be shared with any random person who asks for it” on the consent form and I don’t think it would get through IRB if I proposed such a mechanism. I also don’t think researchers who share their data with anyone who asks for it have properly informed their research subjects of the fact that they do this. I’ve yet to see any solution to this problem, or any significant effort to grapple with the ethical implications of widespread sharing of survey data.
2) My research requires funding to prepare – it’s not cheap. I submit the applications, recruit staff, prepare research documents (sometimes in more than one language), do IRB applications, disseminate advertising and collect data. I also sometimes work with administrative data that I have to pay for. If I hand it over to other people for nothing, and they publish papers off of work that I did the hard graft getting funding for, it’s not very encouraging is it? If I do research in some hot topic area (e.g. see point 3) then I might need large amounts of money to collect the data, and then other people are going to be publishing papers based on my years of hard graft. I don’t think that’s fair, and sure my research should be a public good but now on top of having to pay to publish in open access journals I’m basically paying to provide data to other people. What’s the correct solution to that problem (other than luxury space communism)?
3) Every time climate researchers shared their data they got badly burnt, and the data was used to attack the entire field of climate science. Michael Mann in particular has suffered from this. I am yet to see any mechanism to separate “good faith” attempts to reproduce or check others’ work with bad faith attacks on the entire scientific endeavour. In an era of fascist attacks on vaccination science, virology, gun control, tobacco control, and anything to do with the health of marginalized people, sharing data is just going to lead to worse and nastier attacks on science. It is not going to improve anyone’s wellbeing and it’s not going to improve science.
I’m also concerned about the recent spate of scientists criticizing each others’ work and in particular the slew of reviews of papers that find they don’t share the data. We live in an era when the basic goods established by years of solid, non-fraudulent scientific work – the facts of global warming, vaccination science, the public health of infectious disease prevention, harm reduction, and almost anything to do with the health of marginalized people – is under sustained fascist attack, and the entire field is being accused of fraud and deceit by people who would not recognize honesty if it punched them in the face. When we as scientists do this self-policing in this public way in the current climate, we just feed grist to that fascist mill. I don’t think it helps our credibility or our reputation, and the benefits for science as a whole are marginal since the vast majority of scientists are honest.
1) I think this point is fair. Compromises like synthetic data are better than nothing, but still not great. Maybe we could at least come up with a way to share data with reviewers (and maybe editors).
2) This just sounds like vapid spite. What is the point of going through all that work to gather data, then stubbornly hording it away when it could be put to good use? Maybe if you actually intend to get more than one study out of a dataset, and are afraid of being scooped then, okay, hang on to it until then. But once you are done what possible rational reason would there be to hide it away from that point onward (besides legitimate privacy or legal issues brought up in point 1)? “Luxury space communism”? Really? I’m sorry but this is just childish.
3) Conspiracy theorists (or whatever you want to call them) also use the *lack* of openness to attack science. At least with transparency you can assuage the doubts of legitimate skepticism, and have more ammunition against the less legitimate.
You say that “most scientists are honest”. Transparency is not just about honesty, but even if it was, there’s enough instances of fraud in the scientific community to justify harsher standards of accountability. I mean, read Retraction Watch. Almost every single week they have *some* new story about misconduct. In some cases this involves scientists in very powerful positions given a boatload of taxpayer money. I would agree that it’s not a “majority” but it’s enough. You wouldn’t argue that we don’t need police just because a majority of people don’t commit serious crimes.
DJAD, it’s not “vapid spite” to not want to give my work away for free to others. This was raised extensively during the climate wars of the early 2000s – people worked hard for years to raise money, travel to remote locations to dig ice cores, carefully measure tree rings, spent possibly years doing careful data collection, curation and analysis, and the moment they shared that data the Steve McIntyres of the world misused it to drive home a political point. Being forced to give up material you’ve put hard sweat, graft and sometimes tears into collecting, just so someone else can publish for free, is obviously going to be frustrating. If this became the standard requirement for all data, surely you can accept that this would lead to a class of data vampires, who don’t bother applying for grants or doing experimental research of their own, and just publish papers based on demanding data from other people for “transparency reasons”. The “Luxury space communism” thing is obviously a joke, a short hand for the removal of bad incentives that Dale Lehman is mentioning below.
Conspiracy theorists don’t just use the lack of openness to attack science – they also wage long running campaigns to prove that scientists misuse and misreport data for political reasons. We have decades of evidence of this from the climate wars, when climate scientists (naively, imo) thought that data transparency would solve the problem of denialism, and instead saw their data ruthlessly misused and their good reputation traduced by people who were being funded to defend and indefensible position. We know this is what happens. We know the world we live in. Any calls for transparency need to take that into account. Similarly claims from scientists that we have no obligation to trust other scientists, or these increasingly common gotcha studies about how scientists don’t share their data. We are seeing a crisis of confidence in science at a crucial juncture, a crisis of confidence that has been manufactured by decades of tobacco company money, and we do not need to be adding to it from within.
If you are getting government funding then you are getting paid to produce a dataset that belongs to the people who paid you to do it (ie. The taxpayers) hoarding it is embezzlement and should get you jail time IMHO.
Fortunately NIH and NSF have figured this out and going forward both groups require you to upload your data and to include in the budget the cost of collating and curating and storage etc
https://www.nsf.gov/pubs/2018/nsf18041/nsf18041.jsp#q15
“Vapid spite” is putting it too strongly, but I echo Daniel’s point: assuming you’re talking about government funding then yes I think you should share the data as soon as reasonably possible. You can have first crack at analyzing or summarizing it but research conducted on the government’s dime is supposed to be used for the common good, not for your personal gain. In the U.S. that purpose is explicit and is enforced through the Freedom of Information Act, see https://biotech.law.lsu.edu/ieee/ieee36.htm for example. This is not always how it has played out — some researchers have indeed managed to hold onto their data and to extract money for sharing it — but it’s the goal.
From your use of ‘graft’ in a way that is very non-American I’m guessing you’re from England (here in the U.S., if you say you obtained something by ‘graft’ it means you used bribery or coercion) and I dunno, maybe in England it’s considered fine to take government funds and then refuse to release the data you collect. If that’s the case then I think that in this instance the American way is better.
It’s an Australian usage, perhaps New Zealand, too: “graft” means hard work.
Gee. Was it not a climate scientist who said, “We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it.”
It seems to me that
The authors of research papersClimate scientists have no obligation to share their data and code, and I have no obligation to believe anything they write.The credibility of climate science is an important issue. An unwillingness to share data and code will leave some to reason along the lines of Andrew’s heading for this post.
Bob, I hope you’re not doing that thing of referring to what someone said 20 or more years ago and insinuating that that defines the entirety of a field 20 or more years later!
As John Mashey indicates in his post below, for example, public depositing and sharing of data and code is pretty standard in climate science.
I appreciate these concerns. #1 is a real problem but can be partially solved by deidentifying the data. It isn’t perfect and your example is one of those cases where the concern about publicly providing the data is legitimate. I would only point out that most of the cases don’t fall into that category, but that doesn’t deny that some do.
#2 requires that appropriate credit be given for collection and maintenance of high quality data. I keep saying that that should be rewarded more than the analysis of the data. If the credit were given for curating the data then your concern should go away.
#3 sounds like the complaint against methodological terrorists. Yes, it is a problem, but we can’t go back to the 1950s where trust was automatically granted to those with the right pedigrees. There have been enough bad actors – and innocent bu mistaken actors – that criticism is necessary. After all, that is essential to science. And the idea that if we would just stop criticizing each other the disbelief in science would disappear, is unrealistic. I’ll agree that this is a problem and I’m not sure what to do about it. In fact, I’m quite pessimistic about any scenarios regarding public perceptions of scientific research.
These answers aren’t really satisfactory, and I think that’s because there is no satisfactory way to obtain the transparency some scientists seek.
There is a fundamental asymmetry here: if you don’t trust scientists to be honest with their data, why should my survey respondents trust scientists to be honest with their data? They gave it to me, on the basis that they trust me. On that basis why should I share it? This is a problem not just for the sensitive topics I research, but for anyone who gives data. In my case we sometimes even include a statement in the informed consent to the effect that participation in the research won’t benefit the individual but will be used to improve treatment and policy – they give their data to help improve their community, and if scientists can’t be trusted, I think it’s pretty obvious that *they* wouldn’t want it to be shared. What do we do then?
As for your suggestion for #2, credit is an interesting idea but what does this mean? If someone publishes work with the data I raised money to collect, they add me as an author? What if an economist gets hold of my mortality data (that I *paid* for) and does one of their shoddy little TWFE models with logged rates and normal errors? I don’t want credit for that and if it’s my data I should be able to refuse them the right to publish it, surely? And in this case I haven’t maintained or curated the data – the Australian Institute of Health and Welfare did. I just bought it (with an agreement that I wouldn’t share it, but that’s a different story).
For #3, I don’t know what a methodological terrorist is but I have been threatened – including pictures of myself and my family being released on the internet – because I disagreed with a global warming denialist, who was doing extremely shoddy analysis of a secondary data set. These people aren’t just methodological terrorists – they’re actual fascists. I think it matters what we give them, and as we enter a period of global climate breakdown, with a simultaneous effort across the English-speaking world to exclude transgender people from public life (which isn’t going to stop there), a wave of COVID-19 denialism and a violent, well-funded anti-vaccination movement, we need to be political about the fights we pick. No one in this time period is a pure scientist, we have to be aware of our political context, and in this context we shouldn’t be fueling distrust of scientists. And I would think it’s obvious that demanding transparency, writing papers criticizing others for not sharing data, doing surveys of all the ways in which scientists aren’t perfect, saying “I have no obligation to trust any scientist who doesn’t show me the data”, is fueling distrust of scientists. We can’t stop these people saying and doing what they want to do, but we don’t need to help them. And we certainly don’t need to be giving them our data. You have surely seen what they do with adverse event reporting systems, imagine what they would do with proprietary pharmaceutical company data on vaccines.
I guess a trusted independent agency or organization would be another option, but given that the US right has managed (after a decades-long, tobacco company funded campaign) to ruin the reputation of the WHO, the FDA, the department of education, and is now working hard on libraries, and given their explicitly stated goal of institutional capture (see e.g. deSantis), it would be naive to think any such agency would be safe. No, we are in the middle of a fascist revolution, and we need to act accordingly.
A couple of things you might want to look at: the reference to “methodological terrorist” is an ongoing theme on this blot (see https://statmodeling.stat.columbia.edu/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/). Regarding people’s willingness to trust researchers with their data, you raise valid points. On the other hand, if you look at the NEJM SPRINT competition (https://www.nejm.org/doi/full/10.1056/NEJMp1705323) from a few years ago, you will discover that participants in RCTs were shocked to find out that the data was not being made publicly available.
As for surveys, I find them of little value, so I guess I’m not that concerned about creating difficulties in collecting such data. But the emphasis on data curation that I am suggesting only requires that the powers to be (people that make decisions on promotion, hiring, grants, etc.) decide that curating a useful data set and making it available is worth much more than anybody’s analysis of it. I don’t think your name should go on people that subsequently use the data – regardless of whether they do good or poor analysis – you already got the credit for curating something people found useful. It hurts your publication count, but that is another thing I don’t spend any time worrying about. The incentives are currently terrible, and while I don’t have ultimate solutions, I do think it starts with at least incrementally moving them in better directions (like trying to evaluate the quality of research rather than the quantity).
So you chalk the rampant p-hacking up to incompetence? It is to some extent, and the “system” does select for utterly confused people like Wansink because they don’t need to be dishonest.
But lots of people know what they are doing is wrong and do it anyway “to survive”. Eg, read any biomed paper where sample sizes are reported as “at least three” or random numbers no one would choose like “n=7, n=11”. Stuff happens, but you still need to explain what happened to the other mice.
That isn’t even getting into neglecting to share key details that would totally change someones interpretation of the results. In fact I would say the vast majority of researchers are as weaselly as they can get away with.
And this isn’t new. The original motto was “nullius in verba”, because people were tired of taking the word of incompetent and/or malicious authority figures. Ie, the basis of science is distrust, including (even especially) yourself.
The relatively new thing (~1950) is discouraging testing theoretical predictions (instead now you test a null hypothesis no one believes) and repeating each others work. Those were the safeguards put in place to deal with this issue, they have been largely removed. Now instead we have “trust the science”. That is a step backwards towards a dark age.
And I really don’t get this idea that people are taking advantage of you by reanalyzing your data or replicating your study. In fact, they are doing you a favor. You should feel lucky you don’t have to pay them.
It is like that paper about using prediction markets to replace replications. They outright said the motivation is that most people don’t like doing replications. Well, that means they don’t like science then so why are they in that career?
I’m sorry but people who don’t like replications or sharing data and methods so others can look into it are not cut out for science. It is an incompatible mentality, and there are other careers where it is a better fit.
I understand step-by-step the logic of how you get to that position. But I think that where you’re heading is toward an established Church of The Science, not science.
That’s a reply to faustnotes, in case I clicked the wrong spot.
Faustusnotes, I broadly agree with you and kudos for stating things in a way that could easily be represented – from one perspective – as a deranged viewpoint! The political centre of gravity has unfortunately (and dangerously IMO) shifted so far to the right.
Your point about collusion of (some) scientists in the denigration of modern science – yup, unfortunate but true. On this thread commenters have referred to Wasnick or Susan Fiske, seemingly as exemplars of the sorry state of modern science – these characters come up again and again as if their flaws are symptomatic of flaws in science en masse. “Wasnick => rubbish/fraud => science is broken.” It’s a satisfying conceit and just as Twitter/X has cottoned on that criticism and extreme (often nasty) negativity increases views/clicks so blogs tend to thrive on criticism and negativity. I work broadly in molecular biology and very little of this stuff rings true to me. I haven’t seen so-called “p-hacking” in 30-odd years of research, nor do I understand the negativity against significance testing (it’s all about the interpretation), and any published result of any interest is generally quite quickly reproduced (the “replication crisis” isn’t a “crisis” from where I’m standing tho I can understand why some might find it useful to promote that notion). The identification of rubbish in social science doesn’t mean that all of science is suspect in the way that it may be convenient for some to infer!
On the other hand your thoughts about data sharing don’t match with mine tho I understand how one’s personal experience can influence perspective on this. I consider myself fortunate to be working at a time where all data on, for example, gene and protein sequences, human genetic variants and their phenotypes, protein structures and so on is deposited in freely-available repositories, which provides a fantastic resource. The areas of biomedical meta-analyses and systematic reviews benefits hugely from data availability and I totally agree with comments about the positive aspects of NSF and NIH requirements for data-availability of their funded published research. The Michael Mann episode you refer to is an example of the dangers of agenda-lead attacks on scientific research, but IMO lessons have been learned from this, and in any case Mann ultimately sailed through and continues to make important and productive contributions whereas his detractors are still howling from the margins (on blogs, of course!).
Science is pretty robust and so long as we don’t give too much succour to the barbarians can survive insults from without and within (repeating what I said on my last post)
Thanks Chris, I think, though I’m not sure whether I should be thanking you for the deranged part. Anyway!
I also don’t see much evidence of p-hacking or attempts to be deceptive with statistics. Maybe I’m just lucky. I think people can over-state the role of this stuff, but maybe I’m just immune to some of the pressures that come up in the American system (a student of mine recently got burnt trying to cooperate with a group in NY who were pretty obviously just chasing publications without concern for the consequences, up to and including using chatgpt to “enhance productivity”, but it’s my first experience of this kind of thing I think).
I think a lot of this poor behavior would be reduced by removing the motivation, i.e. the perverse incentives in science. But I don’t want to get laughed at for advocating luxury space communism so I won’t mention that further.
Regarding data sharing, I think it’s much easier to do in the physical sciences because you don’t have people’s wellbeing to concern yourself with, and you don’t have to get trees or rocks to sign consent forms that need to be very specific about some of this stuff. Rather than assuming research on human subjects can simply conform with a single global standard of data sharing, I think it’s important to consider what the consequences would be for data quality and practicality of data collection if we try to impose absolute transparency in every situation. I don’t see any proposals really coming out here that help with this, and i’m not surprised because it’s a difficult topic that a lot of secondary data analysts haven’t thought through.
To answer a few other points here, yes I’m not American and as far as I’m aware that doesn’t disqualify me from commenting on websites. It should be pretty clear from my comments that I am used to working with data from non-American agencies and they don’t follow American rules. As in many aspects of human experience, it might be good to consider whether your US experience is universal, or entirely right. Are Americans the best arbiters of morality in human medical research? Hmmm…
Has anyone tried to do a replication project in your field?
In biomed lots of people claim there is not a problem and they are always replicating each others results in secret unpublished experiments. Then when actual formal replications are done the success rate, even for the very weak criteria of “significant in same direction”, is under 25%.
Eg, you can start here for spinal cord injury research (2003-2012): https://www.sciencedirect.com/science/article/abs/pii/S0014488611002391
Then the next was the “cancer reproducibility project” which ran from 2013-2021). To avoid spam filter I’ll just say you can search for that one. It got a bit of news coverage until covid, when it became inconvenient to the “trust the science” narrative.
Also, from having worked in such lab, talked daily with people in other labs, and reading the literature, I know p-hacking is rampant. And that there are coping mechanisms in play. Like coming up with excuses to avoid replications (eg, not sharing full methods or data) so you don’t have to face how bad it is.
.
In fact you can explore some of the papers with failed replication in the Cancer Reproducibility Project and find that the main findings have been replicated several times, often in different cell lines and organisms. So not “secret unpublished experiments”!
Note that the replication rate was 46% for the Cancer Reproducibility Project: “46% of effects replicated successfully on more criteria than they failed.” if one peruses the project website. However I think you’re missing the point about these studies. They’re not an indication that science is an irredeemible mess, which seems to be the line you consistently persue (and which contains a rather blatant implicit conspiracy theory), but that efforts should be made to make it easier for researchers to reproduce published experiments. Often the problem is that the methods section essentially describes a recipe and that the elements of the recipe can change between publication and subsequent attempts to repeat the methods.
This isn’t a new problem – more than 20 years ago as a post-doc I twice visited labs abroad to learn how to get their methods to work. The value of the reproducibility projects is that they highlight the nature of the problems which can then be addressed. So have more detailed methodology sections; provide videos of experimental procedures; attempt to highlight the essential variables so that, for example, although the original procedure indicates that such and such a reagent is used at a particular concentration, that experiments with different cell lines indicate that the concentration is a variable that should be explored in subsequent studies. In time papers might become editable in real-time so that scientists can indicate, for example, that the culture medium from vendor X no longer works and should be replaced by medium Y.
Your link to the spinal cord reproducibility study says this pretty exactly:
Your last para is anecdotal and not very useful IMO.
That’s 46% of the ~50% of experiments where they could even get enough info to bother with the replication. So total is ~25%.
And cherry-picked indirect replications (it is tried in 20 cell lines then only the one with similar results gets published) are not comparable for obvious reasons that have been known for decades.
It needs to be an attempt to repeat the exact same experiment, so there is no “excuse” available for the failed replication. This was another coping mechanism I saw. Always change up some detail so you get the effect for young male rats, but wait not old male rats. Odd now its not in young female rats either. But the next grant was to check aged female rats, now the effect is back. This isn’t a method thst generates reliable information.
Once we regularly get the same result, then start generalizing.
No that’s not right. I mean – giving a specific example – that a chromatin-mediated drug tolerant state described in a paper looked at by the Reproducibility Project, has subsequently been identified in independent subsequent studies mostly (can’t remember) done in different cell lines. So it’s a robust finding even if the replication of some of the original experiments within the reproducibility project were unsuccessful.
That’s how reproductions are generally done in biomed sci. Unlike some parts of social science where a paper may be something of a stand-alone “gotcha” (“beautiful parents have more daughters”; “big plates make you eat more” or whatever) which are really ripe for direct reproduction, studies in biomed sci, especially pre-clinical ones, are part of the progression of a subject and it’s much more likely that an important observation in one context will be studied in the context of the lab that is following it up. If the general finding is reproduced then that extends the generality of the original observation (or not).
The lowish reproducibility of direct replications isn’t that surprising to me given the “recipe” nature of reported experimental procedures. Two examples:
1. In late summer 1922 Eli Lilly found that their recipe for making insulin didn’t work any more. It couldn’t be reproduced. They didn’t throw their hands up and assert that science is rubbish. The explored the variables and found that the new kind of alcohol they were forced to use by US government wasn’t precipitating the insulin as in their original recipe. So they changed the recipe to accommodate the change of alcohol.
2. A personal one – earlier this year we found our experiments weren’t working because our cells weren’t sticking to the culture plates. After some messing around we discovered the supplier had changed the coating of the plates. Several months later we got it to work again using plates from a different supplier. The paper we published on this some time ago won’t be replicable any more since the original recipe won’t work. We’d hope that someone trying to repeat our work might contact us if they were having problems. If we had a publishing system where we could make real-time edits we could reformulate the methods section appropriately.
This happens a lot..
There may well be differences between fields – I can’t speak about other fields with any authority. I work on public policy issues, mostly in economics. The assumption that most researchers are honest and competent is not something I can support – nor would I claim the opposite. I simply don’t know (my prior would be fairly uninformative). Further, policy issues, by definition, mean that there will be winners and losers. As a result, almost any research must be viewed with skepticism – you’d be foolish not to do so. In most cases, it is not fraud in the sense of falsifying data, but it is more the forking paths problems. Data is carefully chosen and models selected that represent a particular view. “Replication” and “Reproducibility” involve mostly seeing is the results hold up for different data (mostly observational) and/or under different modeling specifications. Lack of data availability renders this process cumbersome and often unproductive. When a researcher has data that I can’t see (sometimes proprietary and sometimes publicly available data that they have manipulated in ways I cannot verify), the subsequent “debate” looks more like people throwing rocks over a fence than direct discourse on the issues.
Perhaps the physical sciences and medicine are different in important ways – I am sure there are such differences, but can’t speak to how important those ways are. But there are a number of research areas where it is wise (I’d say essential) to approach any published study with skepticism and a need to see the data before assuming it is credible. Notably, the examples you highlight (Wansink, Fiske, etc.) work in such areas.
Anoneuoid, I can’t reply to you directly with the weird threading, so I put the reply here.
Actually I have done two replication studies in my field. The first was when Chapman published his paper in 2016 in JAMA showing that the Australian National Firearms Agreement (NFA) reduced suicide and homicide mortality in Australia. Using the data he published with that paper, I showed that if you conduct a proper difference-in-difference model using non-firearm suicides as a control group you find no effect of the NFA. I submitted this to JAMA as a response letter within two weeks of the original publication, following all the due process of the journal, and they rejected it as uninteresting. I subsequently submitted a full paper using data I paid for, and they rejected that too because “we are not a methods journal” so they didn’t accept debate. Eventually that got published in a different journal.
Subsequently another journal – I think Injury Prevention but don’t quote me – published a misleading review of the NFA suggesting it had ended mass shootings in Australia, using an incomplete data set. I responded immediately to the editors asking them to retract the paper since it omitted some mass shootings, and they refused either to do that or to accept a comment (that journal doesn’t routinely publish response letters). So then I constructed my own data set, building on the flawed one (this is all publicly available data), and submitted a full paper which has been rejected from multiple journals because either it’s not original (since someone already published it with flawed data) or it lacks power (an inevitability given the number of mass shootings in Australia).
To be clear I’m a full supporter of gun control but my argument is the NFA hasn’t achieved much, and earlier gun control laws were better, and if we want to end mass shootings we need Japan-style gun laws, which I fully support. I argue in my papers that there is a big risk of making claims about the effectiveness of a law that aren’t correct, since if the USA does implement such a law (haha), it won’t have the promised effects and will damage gun control efforts for a generation.
So yes, I am quite happy to do re-analyses and replications, and to engage in debate about methods. However, it’s a waste of time since top journals don’t publish things they don’t like – see also the terrible ongoing disaster that is the Global Burden of Disease project, the rise of cost effectiveness analysis to the level of an essential tool in health economics even though it’s statistically nonsense, and the early under-estimates of the basic reproduction number of COVID-19 (a journal rejected one of my papers because my estimate was “too high”, and well, how’s that working out?) What I don’t do is accuse other scientists of fraud, assume their data is manipulated, or demand to check their code. In general you should be able to tell the methods people used from their methods section, and from that alone identify whether there are flaws in the study. Which can be done at peer review stage, except I know for a fact (having been involved in multiple peer reviews of, for example, GBD papers) that the top journals don’t pay much attention to negative peer reviews when their mates’ papers are on the line.
The problem in my field is not the transparency of the data!
So your experience is that 0/2 replications got the same result as the original study. Yet you ask us to trust the researchers in your field.
Sorry, but no. It sounds just as untrustworthy as anything else. In fact moreso because these are politically charged issues, rather than something like a solvent selecting for quiescent cells but misattributing the effect to a cancer drug.
Actually, that would become political if people really understood the implications. But at this point they don’t, so there is less risk of bias.
Anoneuoid, strictly speaking they weren’t replication studies, since I am 100% sure that if I had applied their code to their data (or strictly replicated the model they said they used on their data) I would have got the same result. I don’t suspect them of fraud! In the first case I applied a different model to the same data and showed their model got the wrong answer. I could have submitted my response letter without access to their data, since the wrongness of the model was obvious from the methods section of the paper and visual inspection of their figures was sufficient to see the problem, but I added the re-analysis to strengthen the response – a waste of my time as it turned out since JAMA weren’t interested in publishing correct methods.
In the second case the data was wrong, because a bad data set on mass shootings in Oz seems to be floating around (<- this is a side problem with data transparency – bad data sets propagating through sharing, and many re-analyses being done on bad data. Since most people don't check if the data is fraudulent, this will allow a fraudulent data set to infect many analyses. I think there is a recent fraud claim involving a person with a name like Ariely? If they had shared that data 10 years ago then yes, maybe their fraud would have been caught sooner, but more likely many people in many other research groups would have reused the data and produced more flawed results. How do we handle this issue?)
It seems like this flawed data set does not include a mass shooting from 2014 in Western Australia. I don't know why – maybe because it was a family shooting they didn't include it, or maybe it got surprisingly low coverage and didn't enter the data. In any case I think 2 or 3 articles from different authors have used this data. So I updated it to include that data, which then produces a different result. I guess that's a replication? But in any case the papers to date were flawed because they had no control group, so for the paper that can't be published (novelty/power), I collected my own data – redid the Aussie data and included a control trend for New Zealand (a country so close to Oz that its most recent mass shooting was done by an Australian). I don't think it's a reanalysis if you collect your own data and apply a different model, is it?
(As an aside I think different people have different definitions of re-analysis, replication and reproduction and falsification, which also creates confusion).
In any case, as I said, the problem here is not access to the data, it's bad bad methods. I am sure no one involved is committing fraud and I can trust the results they publish. In one case there was a simple mistake, ironically arising from reusing data someone else had shared; in the other case there was a bad model. The reason this isn't more widely known isn't transparency, it's that journals refuse to publish articles that contradict a policy story they support.
Earlier:
I accepted that as correct, but now you say:
So instead of 0/2 successful replication studies you know of, there were none. In that case, we don’t know the reliability the results generated by your field of research.
So, I remain skeptical and will guess your protests about sharing data are a coping mechanism. Because deep down you know it actually less reliable than coming up with an idea and flipping a coin (as is the case for cancer and SCI research).
I am not sure I fully understand the mechanism by which giving someone “my” data to reanalyze, no questions asked, advances science (except in case of gross incompetence or fraud, but then the fraudster might cook the shared data for desired effect). If I am guilty of picking the forking paths that suit me, is that not also true of the next guy? How do we make sure the next guy/gal is smarter, less biased, etc? I think that the implication that the second analyst will rectify the errors of the first is a bit insulting and may explain some of the resistance.
It is different if the person requesting data comes with a new idea, a specific research question, or a variation on the analysis. Then I see no problems in sharing the data. As often as not it ends up being a collaboration.
This is SOO nearsighted it’s laughable. I’ve asked for data multiple times to do things *completely unrelated* to the original reason it was collected. For example once I asked for diving data of elephant seals so I could compare a risk model of decompression sickness. (of course they didn’t give me the data, I have NEVER gotten the data when I ask for it, and I ONLY ask for it when it’s listed as available. it’s NEVER actually available).
I’ve used data from studies on obesity to fit a dimensionless alternative to BMI, but of course that’s only possible when they actually release the data, of the multiple studies I found on the topic only a couple had the data actually uploaded. I sent some emails to the other groups, literally NO response at all.
I’ve wanted data from Pfizer’s vaccine trials because I have a COMPLETELY different decision-theoretic bayesian protocol I’d like to fit and I want to see how rapidly it would converge to an approval. Of course I didn’t even bother asking there is ZERO chance Pfizer will give me access to that.
I’ve also been involved in research by others using publicly available datasets, for example TCGA (the cancer genome atlas) and of course census data like the ACS microdata. Literally thousands of researchers entire research careers might not exist unless such data was available.
In my dissertation I used timeseries data from centrifuge-shaker earthquake models to prove that my equation for soil liquefaction made predictions that were satisfied by the observed data. That was just uploaded as a graph as part of the original paper but if there had been a CSV file from back in 1987 or whatever I could have compared the experimental data directly on a graph in my paper.
There are just zillions of reasons why all data created by publicly funded research should be available or available after de-identification, except in like 0.1% or less of extremely sensitive cases (ie. research on unusual sexual fetishes or on political dissidents etc)
People participating in medical research that isn’t extremely sensitive should simply be required to sign a paper saying “your data de-identified will be released in electronic form in an online electronic archive to further the progress of science and medicine research”. Most research participants already believe this is going on and are upset to find out it isnt!
You seem like a data vampire, Daniel. You let other people collect data and then you analyze it. If you’re interested in doing studies on obesity why don’t you collect the data yourself? The OP’s point is that you can’t trust other scientists’ work – why would you believe the data you’ve fitted your dimensionless bmi model to is not fraudulent? You should collect it yourself. But you don’t – why not? Is it too much trouble? Don’t have the funds? Rather someone else does the hard work? Don’t like people?
There is another model of scientific research in which people collaborate with each other, and as a result of those collaborations people share funding, get access to each others’ datasets, and work together to improve science. It’s not fancy and it requires being nice to other human beings, but it’s also much more productive and depends fundamentally on relationships of trust. It doesn’t involve peppering people you’ve never met asking them to share their labour of love with you so you can hack it for your own papers.
It’s also not embezzlement to not share your data with anyone who wants it – that’s not the definition of embezzlement. I’m not required to let any taxpayer use my govt-funded computer, NASA isn’t required to give away its rockets to taxpayers for free whenever they ask, or let some random bumpkin into their satellite factory to check they aren’t using alien tech. Nonsense. Also presenting decisions by the US government as somehow morally definitive is hilarious. Your govt held a citizen of my country in solitary confinement and tortured him for 10 years. It has nothing to say on what anyone should or shouldn’t do, or on what is right or wrong.
I’ll also point out that everyone here knows Bayes’ rule, so we all know that if fraud is rare, unless the investigators’ sensitivity and specificity is very close to perfect, a large proportion of fraud accusations will be false, with potential career-ending consequences. I’m not singling them out for any reason (just mention them because they’re in the news) but for example do we know datacollada’s sensitivity and specificity? Have they tested their investigative abilities on a wide range of data sets with known properties? Maybe this is why they’re being sued – because if you make accusations of fraud when fraud is rare, you’re inevitably going to get it wrong, and the person you get it wrong about it is going to be very angry. We also know that when fraud accusations fly the seniors quickly push the blame down onto the most precarious people in the workplace. I know US academics are very comfortable with having the majority of their colleagues having very precarious positions, but you should think about the consequences of this for people’s careers, their research activities, their openness and willingness to work with others or share, and even for the future of science as a whole.
Wow! You forgot to mention how horrible gun violence is, how applying Bayes Theorem must lead us to conclude that Trump won the 2020 election, how terrible it is that the 0.1% get away with paying so little income tax, and any other thing you can think of. I don’t pretend to know whether datacolada is right about Gino’s work or not, but the fact that she sued them carriers absolutely no weight in my assessment of the “facts.” Just think of all the people Trump has sued – it must be because they maligned him (and in some way that violates the law).
There are many injustices in the world – and I may even agree with you about many of them. But somehow tying that to “data vampires” is a stretch that boggles my mind.
In anticipation of Andrew losing patience (mine is shot anyway), I won’t respond any more here.
Up til recently of course it would not be considered literal criminal embezzlement. Now that the NSF and NIH have a data sharing requirement and you must in your grant application allocate funds to the data sharing component of your work, and have a specific plan for where and how you will share the data, yes, if you take the funds and then fail to do the data sharing it would be obtaining funds by fraud. I’ll let the lawyers argue over exactly what crime it would be but basically it’d be illegal. This has been needed for a long time. I’m glad it’s finally been put into policy. I’m not sure how well it’s being enforced though.
The purpose of the govt giving Americans funds to do research has never been to further that individual/group’s feathered nest, it has always been to create *public goods* as defined by Economics jargon, that is, goods which are non-excludable and non-rivalrous (ie. you can’t keep people from getting access to them, and once someone gets access it doesn’t eliminate other people’s access).
Of course you can keep people from getting access to your feather-nest of data if you just never release it, but once it’s “out there” it’s easy to disseminate and has virtually zero cost to do so, and copies can be made with perfect fidelity, so there’s no rivalrousness.
Data is pretty much the platonic ideal example of such a thing.
OK. But fyi just in the last year I gave 2 datasets to people unknown to me on request (they had reasonable ideas, imo), and at the end of 2022 I obtained a dataset just by proposing an additional analysis (in ths case I knew one person on the research team). It happens.
Also at least twice I added the dataset as an online attachment to a paper, and not because the journals required it.
I had similar results in trying to get information from medical studies. My emails just vanished into the void—no response whatever.
I was trying to get information about a study that my insurance agency was relying on to deny coverage of an off-label use of a drug. It seemed to me that the study was weak and, given my reading on this blog, I was confident that I could reanalyze the data.
My second inquiry was to the authors of a measurement instrument used in the above study. I had come to suspect that it elicited flawed information. Again, no response. I think I did convince a different researcher not to use that instrument.
I was ready to do my own within-subject study of the drug, when my wife’s employer moved us to a different insurance company that covered the drug and I dropped my inquiry.
It’s disappointing. Based on both (1) various a priori information about the drug which indicate that it should work in this off-label application, (2) comments by practitioners that the drug works in this application, and (3) my own personal experiences with the drug which indicate to me that it works in my case, I am confident that this drug works in this application. But the study is still out in the literature.
Bob76
Bob, let’s face it – you are just a data vampire.
As an applied statistician and social scientist, I’m a data vampire most of the time! Vampiricism is my usual mode of being.
Dale wrote:
“Bob, let’s face it – you are just a data vampire.”
Wow! That’s the highest praise I have received in a long time.
Bob76
Thomas my feeling is that the purpose of sharing data is to enable others to somehow find fraud, not for them to better re-analyze the data. That seems to be implicit in the second half of the title. I think it’s far better for other research groups to collect similar data, analyze it and confirm or dismiss the findings (reproduction) than to do replication of the sort so frequently demanded now. But I think a class of researchers is developing whose research output consists of finding “errors” in other people’s work, and for them data sharing is essential.
In climate science, much data is contributed to public databases, a lot easier with huge decrease in cost of disk storage.
For example, the ITRB (International Tree-Ring Data Bank) collects masses of data from around the world, is heavily used:
https://www.ncei.noaa.gov/products/paleoclimatology/tree-ring
Multiproxy temperature reconstructions (like Jones et al(1998), Mann, Bradley, Hughes (1998, 1999), etc, PAGES2K) only are possible by analyzing selections of large numbers of datasets mostly collected by other people. Modern instrumental temperatures are gathered by people across the world. Accumulated data gets analyzed and reananlyzed numerous times.
For instance, Ray Bradley’s books cover the mechanisms by which signal is extracted from noise, mechanisms that changed over time:
1985 – 472p Quaternary Paleoclimatology
1999 – 614p Paleoclimatology – Reconstructing Climates of the Quaternary 2nd Ed
2015 – 675p Paleoclimatology – Reconstructing Climates of the Quaternary 3rd Ed
Sometimes people find problems with previously-used datasets, sometimes they figure out ways to fix datasets thought too confounded (which was how MBH98 turned into MBH99, they figured out how to compensate for CO2 fertilization of some trees.)
As an observer, I think it’s expected that people who spend years and $$ to gather data (like ice cores) get to publish the first paper(s) on that, with data being published when the papers are … but then others are free to analyze.
If you look at the system/field as a whole, do you really not see the value in making it easier for researchers to detect fraud/errors in the work of other researchers? You write as though you don’t think there’s any underlying truth in these research projects. I sympathize with your frustrations, and there’s a reason why people don’t like the Internal Affairs department, but you must see how your complaints essentially amount to protecting the benefits you receive from the current status quo.
Well, finding fraud is one goal. But reanalysis is another. The last study I did that involved data needed data that a U.S. regulatory agency had developed. It was allegedly available on the web, but I was unable to download it. I just digitized a graph that the agency had published that displayed the relevant data. My analysis showed that the agency was not choosing the least-cost way to achieve its stated goal.
I attached a printout of all my data as an appendix to the study. I wanted to attach a link to the excel spreadsheet. But the lawyers felt that doing so would make it too easy for someone to reanalyze my data. Sigh. If anyone had asked, I would have emailed them the spreadsheet.
Bob76
Thomas:
Your argument is consistent with my above post. If you don’t want to share your data publicly, then no problem, you don’t have to do it! And I have no reason to trust your claims. That does not mean I think your claims are wrong, just that I have no data-based reason to trust them. We’re all fine here.
Oh I agree, nobody has to trust anything.
But I’m a bit surprised that the requirement for trust is to obtain their data to reanalyze according to my own pick of forking paths. It seems inefficient. Also, if I don’t trust their analysis, why do I trust their data?
As for replication, I’m all for it. Collect your own data and analyze them to get an independent result.
The problem is the reward system, I have not yet seen a peer-review process (for funding, publication, promotion, etc) that doesn’t rate originality.
There is no “Reply” button at faustusnotes on September 12, 2023 5:54 AM at 5:54 am. So I am replying here, because I think it is important to clarify where faustusnotes and I agree and where we do not.
The examples of situations that faustusnotes presents are all good examples of sensitive situations where most people would not willingly have that information disclosed. On the other hand, consider a study in which I participated, whereby specimens of succus entericus were withdrawn through a naso-enteric tube to measure concentrations of various enzymes over time in response to certain medications administered intravenously. Not only would I not object to that information being made public, I really cannot imagine anybody who would.
But all of these are extreme examples. faustusnotes correctly makes the point that different people think differently. And I think faustusnotes and I agree that people should be able to act on their preferences in this area. I do believe that participants should be told in advance the extent to which their research data will become available, for what purposes, and to whom. And the investigators’ representations about that should be binding and enforceable. This respects people’s privacy: their right to control the dissemination of their own data. In any study, some decision about how data will be disseminated must be made, in advance, and adhered to. Whatever decision is made for a given study, there will be some proportion of people who will find it unacceptable and decline to participate. Investigators will need to estimate how that will affect both their ability to enroll a sufficient number of subjects and how the selection effect might impact the study results. Presumably, this will lead to information that many people consider sensitive being closely held, and information that many people wouldn’t care about being widely available. This is true privacy, and I support it.
What I object to is the current practice, which is not privacy, but secrecy. By default, all information is closely held, with no consideration of what the consequences of wider dissemination would be, nor of whether participants would be agreeable to dissemination. This is, in fact, *not* respectful of people’s privacy, because in the case of non-sensitive information that they are willing to share, by preventing that sharing, current practice degrades the usefulness of the data that people like me give their time and effort to create.
I would like to see practice move to a more flexible approach that treats different kinds of data differently, and, of course, leaves the last word to the potential research participant.
Clyde, kudos to you for participating in that study, it sounds awful.
I think a problem with giving participants an individual choice about how their data is used is that it will mean any data set that is shared publicly is not complete, making replication impossible and increasing the risk of bias and accusations of fraud. The sharing decision has to be made at study level, not individual level. I think you’re right that there are many topics where people will mostly be willing to have their data shared (if properly de-identified) but this isn’t always the case, and a blanket transparency rule will lead to some studies just being impossible to do because research participants will refuse at the point of consent.
this is doubly complicated by the fact that so many people have so many different ideas about what is sensitive information and what is not, and what is identifying and what is not.
For example I’ve done a lot of research fieldwork and absolutely the toughest was a study of sexual behavior in pregnancy among heterosexual women, conducted as part of a scoping project for a herpes vaccine. Even though it was clear that the data would never be shared, was completely de-identified, and would be treated with the utmost confidentiality, pregnant women simply refused to have anything to do with that study. Huge rejection rates just for asking about their sex lives before they got pregnant in a two page, completely anonymous and confidential questionnaire. Everyone involved in the study was shocked at how reticent the potential subjects were. I am convinced that if we had told them their data might be shared with other researchers we’d have got a close to 0% response rate. And of course if we had offered the option to share data, only the purest of the pure respondents with nothing to hide would have agreed to that.
I don’t think there’s any way to balance the transparency concerns people have with the impact of these concerns on the practical realities of fieldwork.
@Chris
That paper is actually a great example of unreproducible in principle. They couldn’t even get past the first finding, which was the premise for the rest, because we have no idea what the cells were even treated with. From the supplementary methods:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2851638/
Erlotinib isn’t water soluble, so 2 uM most likely doesn’t mean dissolved in water. There is some other solvent or mixture of solvents being used, typically some mix of DMSO + H2O. DMSO is also well-known to be biologically active with its own dose-response curves. Eg,
https://www.nature.com/articles/s42004-020-00409-7
So the replication team first tries to figure out the DMSO concentration (assuming that is what was used), and couldn’t get the same results using 0.01% or 0.1% DMSO. However, with 20 uM (10x more) Erlotinib in 0.1% DMSO they could get roughly similar results.
Then they started thinking maybe the hydrocholoride salt was used instead of freebase Erlotinib (which would be more water soluble), but still couldn’t match the results. This is like the difference between crack and powder cocaine, which have very different effects.
I would have tried 1 and 10% DMSO, but sounds like they started thinking this was a waste of time and moved on:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8651290/
This type of thing is exactly why we need regular, published, direct replications. Just because someone else found some formulation of Erlotinib that had a similar effect on some cells doesn’t constitute a replication. The opaque methods rendered the results basically a waste of money, the paper amounts to an interesting idea.
This replication project data is actually awesome. So I was wondering on the DSMO effect, and they actually share all the data:
https://osf.io/9g2kc
And yes, once you get to 2% DMSO only ~50% of the PC9 cells survive. I would bet the original paper used some high concentration of DMSO in the range of 1-10%. Then like 50-90% of the supposed “chemo drug” effect was actually due to a DMSO artifact.
My favorite thing about open data is the thought that someone (in the future?) might follow a couple of data-sets and look at the papers that used that data-set to perform statistical analyses with.
This someone might then note, for instance, that data-set X has been used by 22 papers thus far, and subsequently ponder about the validity of the statistical findings in these 22 papers given the (likely?) possibility that none of these subsequent p-values have been controlled for multiple comparisons.
The possible resulting discussions concerning whether or not there is a real and valid difference between Professor P. Hack performing multiple analyses without controlling for them at a certain point in time, and the situation where different scientists perform multiple analyses on the same data over time would possibly be an interesting (and/or amusing?) sight to behold.
If each of the 22 analyses are very similar then the Type I error rate won’t truly compound enough for conventional adjustments to work. If each of the analyses gives you roughly the same result then adjustments are redundant, regardless of how different they are.
If it’s a garden of forking paths situation, where each analysis is doing a completely different thing and arriving at a completely different result, then the discussion should focus on the overall rationale of each analysis. If any one approach is clearly better, to the point of making the others look obsolete, then the other p-values become irrelevant as they were computed under sub-optimal rationale (put aside the fact that some argue that significance tests are an *inherently* sub-optimal form of inference to begin with). Of course, not everyone will necessarily agree on what the “most robust approach” is, I’m just saying that whatever discussions do arise out of data re-use really shouldn’t revolve around p-values.
Either way, I doubt multiple comparison adjustments will play any major part of any discourse that comes out of data re-use. And we’re of course ignoring the (very likely imo) possibility that a lot of these 22 analyses won’t even use p-values for inference.
I’m trying to understand what you mean, and its implications.
In the context of possibly adjusting for multiple comparisons, would you say that the situation of Professor P. Hack performing 22 analyses on data-set X at a certain point in time is or is not effectively the same as multiple scientists performing the same 22 analyses on data-set X over time?
What’s much more likely is that bad data sets will be reproduced and reused across multiple researchers, because while all the people commenting here think they’re the smartest people in the room, the reality is that finding fraud is hard and the likelihood of looking for fraud is directly related to how prestigious the fraudster’s institution is. This is why the R&R error took so long to find, while they were wandering the world convincing governments to cut welfare to the poorest. It’s also why that recent scientist is comfortable suing datacolada, and why the Ariely (?) fraud took so long to uncover. Most people don’t look, and those who do look are as likely to make a false accusation as to prove anything (Bayes’ Rule).
There’s no reason to think transparency will improve science – maybe it will just make a bunch of bad data more readily usable. Dale Lehman (I think) up above said he doesn’t think much of the value of survey data. Well then why do you want it to be transparent? So whole generations of grad students can use it for studies you fundamentally don’t trust (you’re wrong of course – survey data is very useful – but you should examine what you’re asking for here).
It’s amazing that in the era of Trump and Musk – guys who are obvious frauds, like strutting around in public saying “I’m a fraud” with glowing signs above their heads saying “this dude’s a fraud!” – when so many people believe these guys are honest, serious people, you guys think that somehow scientists are going to bust other scientists’ frauds if you can just see the data. Isn’t the first sign of a mark their overconfidence in their ability to see through the trick?
What I have started to wonder is whether it would be possible, and how easy it would be, for some computer-savvy folks to start producing fake data-sets which are impossible to spot or mark as possibly fraudulent due to them being sophisticated enough, or whatever.
It’s like taking the Diederik Stapel fraud to another level!
Given all the computer power and possibilities, and the possibility of the usefulness of certain data and its conclusions and implications, I wonder what can even be trusted anymore…
I produce thousands of fake datasets all the time for use in exploring the implications of priors in Bayesian models, sample size analysis via simulation, etc. Due to the use of random number generators from different probability distributions and hierarchical nature of the models being explored, I doubt anyone could distinguish these from ‘actual collected’ data. For simulation, the whole goal is to make the fake data as ‘real’ as possible and explore the effect of changing different parameters. This has been relatively easy to do for years now. It’s a very powerful and useful tool to aid in model specification and study planning.
I don’t think you need to be a tech wiz to do this – there’s huge amounts of e.g. demographic and health survey (DHS) data available over many years and countries from which you can splice together information, change variable names and produce plausible datasets with no relationship to reality.
In contrast to this, I think it’s also quite likely that real datasets could look fraudulent to people unfamiliar with them. This was the point I was raising about Bayes’ rule above (which got dismissed) – if fraud is rare, and our ability to detect fraud when we try is not perfect, a large number of false allegations will be made. We’ll spend all our time pointing fingers at each other, and getting nowhere.
1. I am not saying that open data will solve all problems with empirical work.
2. I am not saying that fraud is the most serious problem with empirical work.
3. I am saying that I don’t find survey data very useful. I’m sure there are exceptions, but the fundamental issues involving the fact that survey respondents have little incentive to think carefully or truly about their answers, combined with the difficulty of clearly wording surveys to get meaningfully measured responses, make most surveys less than useful to me. You claim that survey data is “very useful” and I would debate that (though not here since it will take forever and Andrew will lose patience before either of us will convince each other of anything) for most uses of survey data that I have seen. In many cases, I think they do more damage than good.
4. The best case for transparency improving science is the potential for deterrence. I do think people will be less likely to “torture their data until it confesses” (I believe that was attributable to Ronald Coase) if there data is readily available. Will it solve issues of poor and/or fraudulent data work – certainly not. Will it improve the practice of data work – I believe so.
5. I certainly don’t think I’m among “the smartest people in the room.”
6. I’m not so sure that the R&R error and the Ariely fraud “took so long to uncover.” Once the data was available, issues with it became apparent fairly early on. What took so long was for the errors to have any impact – this has much to do with institutions and incentives, not the difficulty of finding these errors. Not to minimize that – some of the detective work is quite ingenious.
7. I have no idea how your tirades on Trump and Musk relate to these issues. You might as well add Santos, Marjorie Taylor Greene, and the majority of politicians to the list – and I am in agreement with you regarding the frauds they represent. I am at least distressed as you are that so many people apparently believe these people to be honest and serious. But tying that to open data is a stretch – and not helpful in my opinion.
Finally, I want to emphasize that fraud is important, but it is not the main reason I support open data. Data is important to many policy questions – and it appears that you have been involved in some of these. So have I. But when each study uses different and proprietary data and reaches different conclusions, it is hard to see the path forward. I’ve been in many regulatory “battles” and the role of data is undermined by this lack of transparency. It results in decision makers discounting the value of the data, and resorting to pedigrees and mood affiliation in deciding what analysis to “trust.”
Dale I raise the Trump and Musk “tirade” (one sentence!) because they’re classic examples of obvious frauds that fool many people. The Musk one in particular – so many times over the years before he bought Twitter so many people were so sure he was so smart and such a business genius, when he *obviously* wasn’t. There’s no reason to think that we – or people eager to uncover fraud in data – are necessarily better at their task than all the journalists who were supposed to uncover Musk’s obvious terribleness but just didn’t. It seems like we in modern society just aren’t very good at this. See as well my point above about Bayes’ rule – it doesn’t help science as a discipline at all to have false accusations constantly being flung around!
If your concern is not with fraud, then I think a lot of what you are concerned about can be solved by institutional changes on the one hand (as you’ve mentioned before) and by a culture of clear article writing. In my re-analysis experiences, the problems were obvious from the methods section of the paper, for example. As an exemplar of bad article writing, consider a typical econometrics paper where a) methods and results are mixed together, b) lots of details of data and methods are buried in footnotes or figure legends, c) many components of regression models are not presented (just written as “adjusted for …” in a footnote) and d) confidence intervals are not presented (just SEs). I am convinced that there are a lot of LPMs published in the econ literature that clearly allow for negative probabilities, but it’s simply impossible to tell because the constant of the model and the CIs aren’t clearly shown. Also when do any of those papers ever mention residual analysis? If those articles followed a clearer, more scientific structure, and presented all their results in a more digestible way, we would see – without needing access to any of the data – that they were badly done.
In my view you should be able to understand what was done and infer whether the model is good or bad simply from the methods section of the paper. Sure not all details and possible errors will be caught there, but the broad structure of a lot of scientific work would be improved with clearer writing. Which we can all train people in, and assess during peer review!
I also think a lot of fraud and mistakes occurs in labs fraught with bullying. Better institutional protections for junior staff and students will cause a lot of these problems to evaporate – the rest of the institutional changes you’ve already hinted at and don’t need repeating.
And speaking of institutional changes – if this libel case against datacolada succeeds for the plaintiff, we can all kiss goodbye to any future fraud investigations of any kind!
We are in substantial agreement. The main area I would dispute is the “solution” of improved writing of methods. I’m all for that, but I don’t think it can come close to what is needed. Better writing of methods may well clarify for me whether a paper has inappropriately analyzed the data – but the world I work in requires that I be able to document what effect it has, not just assert that the methodology was inappropriate. On a witness stand, such assertions are worthless, whereas demonstrating exactly how a more appropriate analysis differs can matter. Similarly, provision of the code used is often suggested as a remedy and it would certainly help. But it isn’t easy to work through someone’s code and it would need to include all the processing of the data that was done. Forking paths have a way of avoiding recognition.
I also think you are making too much of the datacolada case. I don’t pretend to know whether the suit has merit or not, but it carries no weight in my mind that a suit has been brought. Trump has sued many people and unfortunately some people take that as “evidence” that there was something done wrong. Gino has sued – do you mean that her defense (which we have not yet seen) has merit? Surely, it will take more before I am prepared to believe it has any merit.
You do seem to have a prior belief that most research is sound, that fraud is rare, and that most accusations are “false.” I wish I could share your view, but I don’t. I won’t go so far as to declare the opposite, but given all the institutional problems (poor incentives, etc.) I am agnostic. In any case, I do believe the more serious problem is poor analysis, not fraud, though I think much of this is intentional, not accidental. By intentional, I mean much analysis is done until the desired answer is obtained, regardless of how tortured the analysis must be to get that.
Dale this is a response to your point below, due to the threading.
I don’t have a “prior belief” that most accusations are false. That’s my posterior conclusion based on applying Bayes’ Rule to a discrimination task where the event in question is rare. My prior belief is that fraud is rare, yes, and thus my conclusion is that many accusations will be false.
I think the datacolada case is the first instance of someone being sued for alleging research fraud, at least that I’ve ever heard of. And I think the researcher’s response to datacolada is probably making some people think carefully about whether to accuse someone of fraud. I don’t think that’s making too much of an issue.
I agree that it is often difficult to describe the exact effect of a wrong method without having access to the underlying data, but given the lack of willingness to debate methods that I have experienced, I’m not convinced having the data will make that much difference. And I agree that reading others’ code is not sufficient to answer that question. I also suspect that in some branches of some fields there is no code – it’s all menu-based.
Then of course there’s the image manipulation problem in some parts of biology. That’s even nastier, because as far as I know there is no underlying data to check …