Every once in awhile we receive data or code requests. For example, this came to Erik van Zwet, Simon Schwab, and me, in regard to our 2024 paper, A new look at p-values for randomized clinical trials:
I hope this email finds you well. My name is Boyao Li, and I am a 4th-year PhD student in Biostatistics at Duke University. I am currently working with Daniel Ting at Meta on research related to effect size distribution estimation.
We are very interested in your work on the Cochrane dataset and P values of RCTs, and I am currently exploring the CDSR dataset mentioned in both the statistical property of RCT paper and the P values for RCTs paper. As part of my analysis, I am trying to reproduce your research using the CDSR.RData shared on Open Science Framework.
However, I have encountered some issues when attempting to extract the relevant data from the R dataset. Specifically, the dataset contains effect size estimates and related z statistics from 34,773 RCTs (RCT=yes), which exceeds the 23,551 RCTs mentioned in your papers. Despite following the instructions in the reference, I’m having trouble selecting the correct data to match the distribution plot of observed z statistics shown in the RCT stat property paper.
Could you please provide guidance on how to filter the data correctly? I’m particularly unsure about how to select among different classes in the column outcome.flag (DICH, CONT, IV, IPD) and outcome.group (efficacy, safety, bias, dropouts). Any additional references or advice would be greatly appreciated.
Zwet fielded it:
Hi Boyao,
On the OSF page there are two datasets: CochraneEffects.csv and CDSR.Rdata. You can get those 23,551 RCTs from “CochraneEffects.csv” by running the following R code:
set.seed(123) # for reproducibility d=read.csv("CochraneEffects.csv") d=d %>% filter(RCT=="yes") d=d[d$outcome.group=="efficacy" & d$outcome.nr==1 & abs(d$z)<20,] d=group_by(d,study.id.sha1) %>% sample_n(size=1) # select single outcome per studyWe added CDSR.Rdata at a later time. It’s basically the same, but it has a few extra variables. There are also some slight discrepancies because you run the following code, you get 23,557 RCTs instead of 23,551:
set.seed(123) load("CDSR.Rdata") d=data %>% filter(RCT=="yes") d=d[d$outcome.group=="efficacy" & d$outcome.nr==1 & abs(d$z)<20,] d=group_by(d,study.name) %>% sample_n(size=1) # select single outcome per studyGood luck with your research, and don’t hesitate to let me know if I can help.
Erik
I wanted to share this exchange, just as an example of how even the easiest data requests can require some effort. This sort of experience motivates us to post our data and code ahead of time to avoid these later difficulties for us in sharing the data and for later users.
More importantly, think of all the users who are too shy to send emails to the authors! It’s best to post the data online so anyone can download. None of this “Data available upon request” (or, worse, “Data available if you can give us a good reason why we should share with you” or, even worse, “Data unavailable”) crap.
Data-sharing is important both to our replication of our own work, but also to advance other projects. The public usually funds our data collection and we should view it as a public resource.
That said, there are serious challenges related to the lack of shared norms for how this should work. In my career I have experience the following issues: (1) I give a talk for a paper that has not even been submitted to a journal. Someone demands my data immediately to write a paper that will probably “scoop” me. (2) This has happened with grad students at other institutions: they ask for data and I send it, or they just download it from my website. Then they try to enlist me as an unpaid research assistant: lots and lots of questions that could be answered if they would just read the research that used the data. (3) (Yes, this really happens): Someone downloads data from my earlier projects, and then expresses annoyance that I only assembled data on what I needed for that project. They expected that I would have lots of other data relevant to their research but not mine. One grad student actually went so far as to ask me to collect other data they needed for their work.
I hear of more serious issues. The challenge for all of us is to recognize the importance of openness to data-sharing, but not kill incentives to assemble new datasets.
“The challenge for all of us is to recognize the importance of openness to data-sharing, but not kill incentives to assemble new datasets.”
Back in the beginning days of all this openness and transparency and incentives-stuff I had the idea (likely inspired and influenced by something I saw on a website from someone) to provide some opportunity to people who, for instance, make their data available to make this clear on their Curriculum Vitae (CV).
I reasoned that that could perhaps be seen as being in line with transparancy, making things more clear, making a distinction between things (just like making a distinction between exploratory and confirmatory analyses), and perhaps it might even be(-come) an “incentive” for people because they can hereby make more clear to possible funders, members of PhD committees, and others that they made their data available via an icon or badge depicted on their CV.
For those interested, here is a link to a google group discussion board where I mentioned the idea back in 2016, and certain things occured that can be read there that still puzzle me to this day. Although, I must add that they puzzle me much less after all that has happened inbetween regarding such things as the possible hyping of open practices badges (see the blogpost by Hilda Bastian, 2017 titled “Bias in open science advocacy: The case of article badges for data sharing), the unavailability of pre-registration information concerning Registered Reports (see Hardwicke & Ioannidis, 2018), and the more recent case of the problems with pre-registration in a study that received lots of attention if I’m not mistaken (see Bak-Coleman & Devezer, 2024).
https://groups.google.com/g/openscienceframework/c/7oeul9pusKg/m/AEMHamSxCgAJ
(The idea in the form of a manuscript can now be found on SSRN with the title: “Open Practices Badges for Curricula Vitae: An Additional Way To Help Change Incentives In Psychological Science?”)
2004 or 2024?
Typo fixed; thanks.
The paper is actually from 2023. The published version is here
‘None of this “Data available upon request” (or, worse, “Data available if you can give us a good reason why we should share with you” or, even worse, “Data unavailable”) crap.’
…or ‘No data was used’ in an empirical analysis!
The linked PDF is from “2 October 2023” not 2004.
Around 90% of the data I use, I don’t “own”, it’s government data on people. I can’t share the data because it’s not mine to share. I can tell people how to access the data.
Another 5% of the data I use, I obtained permission from the copyright holder to use. It’s also not mine to share. (My country has stricter rules around “fair use” for copyright data and I probably played (slightly) on the safer side). I can tell people how to access the data.
The remaining 5% I do “own” but I can’t share because of ethics restrictions around consent – we never tried to obtain consent for “further research purposes” because it would have depressed participation and my impression is that it would have been a nightmare to get through ethics anyway. We generally have a condition in ethics that we delete the data 7 years after end of the study (IIRC). So, the data could be deleted by the time someone asks for it.
There are genuine reasons why people can’t share data.
In 90% of cases, it is very likely that the data IS actually re-shareable, depending on the license it uses. It would be relevant to do so, as sometimes data sources changes and it is crucial to capture the state of the data at the time of analysis. If not, you can at least provide a script that access the data in a standard way to allow for reproducibility. Few people bother.
In 5% of cases, although the license is restrictive, some by-products of it (summaries, model outputs, basis for figures) are shareable. Again, few people bother.
In the last 5% of cases, I find it hard to believe that obtaining consent for anonymised data is that hard and restrict response rate that much. But in the end, it’s also an epistemological choice: do we believe we are doing the right kind of science when we can’t appropriately convince our peers of our conclusions? (Which is only possible when data is shared along with code). Few people bother to care about that epistemological choice.
You can’t really go over one, well spaced, a4 page of information for consent before people start bugging out. The people we were surveying were highly intelligent but very, very busy people; extra impediments reduce the response rate.
In a large (~12,000 participants) gen pop health survey, around 10% (IIRC) of people declined to allow their data to be used for further health research (even after being told all their data would be anonymised). So, in an all or nothing consent situation, typical in online surveys, that matters.
I haven’t seen any license that lets you re-share unit record data about people. What is the point of the license in that case? They could have just published the raw data online in the first place.
I’m sure most statisticians do write scripts to recreate the data and analysis. Otherwise responding to reviewers becomes a nightmare.
In my particular case it wouldn’t matter – new instances of the data are created every 3 months and (mostly) deleted a year later. By the time a paper is published, the data has most likely been deleted.
In the end sharing data doesn’t guarantee that the research was done well. It’s no guarantee that the data haven’t been screwed with. I’m pretty sure Boyao Li didn’t look at the 23,551 RCTs to see if Andrew (or who ever) copied the data faithfully, he took it on trust.
I’d like to point out just for peoples background info, that in the US there is no such thing as a copyright on datasets, this is well decided case law from the 1990s when someone tried to copyright a phone book. There is therefore no such thing as a “license to use” a dataset. You can have a contract that a person will give you some data if you dont further share it but this is not enforceable under copyright law specifically.
Anon:
Yes, if data are public, it’s fine to give instructions on how to find them.
If I were an author on this paper, the main story for me would be that the big mistake here was the way in which the data was released. I would want to learn something valuable from my mistakes here. The code is apparently (see below) in the appendix; how much more work would it have been to produce a reproducible R Markdown file that takes the user from the beginning to the end of the analysis? Just releasing a csv and Rdata file is what led to the query in the first place. What’s the reader supposed to do with raw data files? Trying to reconstruct an analysis from scratch is just wasting people’s time.
I’m constantly surprised by the lack of consideration that authors have when they release data with their papers; just put yourself in the outsider’s shoes. I have made this mistake in my data releases in the past, I learned only slowly to be more considerate to the outsider.
In the files section of the osf, there is no README file. I would have spent some effort in explaining what’s in these files, not wait for someone to contact me.
Also, in the section called Data Release and Availability or whatever it was called exactly, I would have provided the link to the osf repo, instead of pointing the reader to a reference at the back of the paper, which contains the link. For me that was a very strange way to point the reader to the data, especially in a section called Data Release or the like. Just release the data right where you have the section.
Finally, the Data Release section says:
> R code to reproduce our results is provided in the Appendix
Uh, what appendix? There is no appendix in the paper. Am I supposed to search the journal’s website to find it? I went to the NEJM website and searched for appendix, but could not find it anywhere. I did see some references to a Supplementary Appendix twice, but no link to it. And interestingly, in the published paper, the data availability statement says cryptically:
> A data sharing statement provided by the authors is available with the full
text of this article at NEJM.org.
They actually removed the reference to the osf repo in the published paper! Then I clicked on the linked NEJM.org page. That just took me to https://www.nejm.org/. How much harder would it have been to link to the actual data release statement for this paper? I eventually found the data sharing statement, which does not even remotely resemble what’s in the draft that Andrew linked to:
https://evidence.nejm.org/doi/suppl/10.1056/EVIDoa2300003/suppl_file/evidoa2300003_data-sharing.pdf
Then the published paper also says:
> Author disclosures and other supplementary materials are available at
evidence.nejm.org.
I thought, finally; now I’ll get to the supplementary materials. But no. The linked website is a generic website. I now have to search for…what exactly? I searched for Gelman Greenland and was thrown back to the main website containing the article. There I googled for supplementary and found the R code finally:
https://evidence.nejm.org/doi/suppl/10.1056/EVIDoa2300003/suppl_file/evidoa2300003_appendix.pdf
How much harder would it have been to just link to the osf repo in the Data Release statement and to put in the code, as R Markdown (it’s clearly written in LaTeX, so I assume there is R Markdown or an Rnw file behind this), as well as the compiled pdf?
Conclusion: The big story here is that the data release was totally inadequate and we should do much, much better. Actually, the fact that even the big names in statistics are not releasing data and code in a usable way makes me realize that it’s hardly fair to expect others to do a good job. We have a long way to go yet in open science.
I found it really odd that the key takeaway by Andrew here is that even the simplest query can take effort to respond to. In fact, the key takeaway is that the data release was totally inadequate.
Shravan: The supplemental file with the R code which you found by Googling is actually directly available from the NEJM Evidence website. Just scroll down to Supplementary Material.
The analysis of those 23,551 trials was done in an earlier paper from 2021. For that paper, I should have provided the code snippet that extracts those 23,551 trials from the larger datafile.
Hi Erik,
My main point, which probably got obscured by my ranting on and on, was just that everything could easily be placed in the osf repo. Rmd file containing fully reproducible code, the corresponding pdf, the data (which was there), and a README.md that explains what’s in the data and how to use the repo.
My general irritation stems from too often being unable to access other people’s data+code from their published papers. My own earlier papers had this problem (data and analysis not properly documented), and probably I am unable to get this completely right even now. These days I am struggling to reproduce some published analyses (not the present work under discussion), something that could have been avoided if the authors were just to provide their data and code in a usable way. It’s a great pity we still don’t have any standards for this, the way we do for section headers in psych-type journal articles.
Hi Shravan:
Broadly speaking, I think there are 3 kinds of readers of scientific papers:
1. Those who just take the authors’ word for it. I’m afraid this is the majority :(
2. Those who want to see how the authors obtained their results. For these readers a nicely formatted analysis report with explanations, code and output will be most helpful. It’s really the ultimate “methods section”. A rendered Rmd (pdf, html or docx) is a great way to make such a report. I think it’s best to publish this report together with the paper on the publisher’s website. The supplement of “A new look at p-values” is an example.
3. Those who want to run the code themselves, maybe make changes or try new things. For these readers, the Rmd file would be helpful so that they don’t have to copy-paste from the analysis report. In practice, however, authors typically provide a big folder on OSF or Github with lots of separate R-files. That may be good programming practice, but it’s not very helpful for type 2 readers (which is typically me).
Publish the data with the paper, even ~1900 you can find papers that included tables of the raw (not summarized) data.
If you can’t for whatever reason, legit or not, it puts the entire project into question. That violates the nullius in verba principle. We simply should not trust people to analyze their data correctly. And especially not ourselves.
If you find yourself lacking a deep desire for others to reanalyze the data or, even better replicate the entire process, something is wrong. The raw data is the most valuable part of the study you created.
Yes. I keep saying that we need to change the incentive structure to reward cultivation, documentation, and provision of interesting data and downplaying the analysis of such data. Analysis can be easily automated (yes, abused at times, but often competently done by software) but data quality not so much. The rewards structure in academia reflects the world decades ago when the hard part of analysis was the methodology. Analysis methodology is not trivial, but I think the more critical tasks are measurement, documentation, cultivation, and presentation of data – tasks that generally receive little to no credit.
The analyses can also be performed in parallel, but it requires the initial data to be collected first. That is the bottleneck. It is like our species is missing out on 10-1000x efficiency opportunity.
This is exactly my experience in ecology. It’s increasingly trivial to perform complex analyses, especially given the ability of AI to produce code; the difficult part now is what you say: accurate measurement, documentation, interpretation. So much nonsense is hidden beneath complex analysis, “storied” for impact.