It’s a workflow thing.
Here’s the story. Carlos Ronchi writes:
I have a dataset of covid hospitalizations from Brazil. The values of interest are day of first symptoms, epidemiological week and day of either death or cure. Since the situation in Brazil has been escalating and getting worse every day, I wanted to compare the days to death in hospitalized young people (20-29 years) between two sets of 3 epidemiological weeks, namely weeks 1-3 and 8-10. The idea is that with time the virus in Brazil is getting stronger due to mutations and uncontrolled number of cases, so this is somehow reflected in the time from hospitalization to death.
My idea was to do an Anova by modeling the number of days to death from hospitalization in patients registered in 3 epidemiological weeks with a negative binomial regression. The coefficients would follow a normal distribution (which would be exponentiated afterwards). Once we have the coefficients we can simply compare the distributions and check the probability that the days to death are bigger/smaller in one of the groups.
Do you think this is a sound approach? I’m not sure, since we have date information. The thing is I don’t know how I would do a longitudinal analysis here, even if it makes sense.
My reply: I’m not sure either, as I’ve never done an analysis quite like this, so here are some general thoughts.
First step: Plotting the data
Start by graph the data using scatterplots and time-series plots. In the absence of variation in outcomes, plotting the data would tell us the entire story, so from this point of view the only reason we need to go beyond direct plots is to smooth out variation. Smoothing the variation is important—at some point you’ll want to fit a model, I fit models all the time!—; I just think that you want to start with plotting, for several reasons:
1. You can sometimes learn a lot from a graph: seeing patterns you expected to see can itself be informative, and then there are often surprises as well, things you weren’t expecting to see.
2. Seeing the unexpected, or even thinking about the unexpected, can stimulate you to think more carefully about “the expected”: What exactly did you think you might see? What would constitute a surprise? Just as the steps involved in planning an experiment can be useful in organizing your thoughts even if you don’t actually go and collect the data, so can planning a graph be helpful in arranging your expectations.
3. A good plot will show variation (any graph should contain the seeds of its own destruction), and this can give you a sense of where to put your modeling effort.
Remember that you can make lots of graphs. Here, I’m not talking about a scatterplot matrix or some other exhaustive set of plots, but just of whatever series of graphs you make while exploring your data. Don’t succumb to the Napoleon-in-Russia fallacy of thinking you need to make one graph that shows all the data at once. First, that often just can’t be done; second, even if a graph with all the data can be constructed, it can be harder to read than a set of plots; see for example Figure 4.1 of Red State Blue State.
Second step: Statistical modeling
Now on to the modeling. The appropriate place for modeling in data analysis is in the “sweet spot” or “gray zone” between (a) data too noisy to learn anything and (b) patterns so clear that no formal analysis is necessary. As we get more data or ask more questions, this zone shifts to the left or right. That’s fine. There’s nothing wrong with modeling in regions (a) or (b); these parts of the model don’t directly give us anything new, but they bridge to the all-important modeling in the gray zone in the middle.
Getting to the details: the way the problem is described in the above note, I guess it makes sense to fit a hierarchical model with variation across people and over time. I don’t think I’d use a negative binomial model of days to death; to me, it would be more natural to model time to death as a continuous variable. Even if the data happen to be discrete in that they are rounded to the nearest day, the underlying quantity is continuous and it makes sense to construct the model in that way. This is not a big deal; it’s relevant to our general discussion only in the “pick your battles” sense that you don’t want to spend your effort modeling some not-so-interesting artifacts of data collection. In any case, the error term is the least important aspect of your regression model.
Third step: Using graphs to understand and find problems with the model
After you’ve fit some models, you can graph the data along with the fitted models and look for discrepancies.
Fourth step: Improving the model and gathering more data
There are various ways in which your inferences can be lacking:
1. No data in regime of interest (for example, extrapolating about 5-year survival rates if you only have 2 years of data)
2. Data too noisy to get a stable estimate. This could be as simple as the uncertainty for some quantity of interest being larger than you’d like.
3. Model not fitting the data, as revealed by your graphs in the third step above.
These issues can motivate additional modeling and data collection.
The days to death *are* bigger/smaller in one of the groups. We don’t need any data to tell us this. In fact if the data showed the exact same value, we can be confident that is just a coincidence that won’t generalize.
So if that is truly the question you want answered, we learn nothing from this analysis.
Of course, no one actually cares about the answer to that question. I’d guess what you actually care about is understanding the dynamics of viral transmission and infection in order to predict the future (and anticipate the effects of any interventions).
Where you want to start is a SIR model: https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology
You will also want to model the dynamics of testing, treatment, reinfection, behavior, etc. But the basic idea has been explored for a century.
And you will probably find there are just way too many unconstrained parameters involved to really tell what is going on. That is a perfectly fine valid conclusion, it is useful because we have gained info on what types of data should be collected in the future. You need to check the model’s predictions on new data anyway, not the same data it was devised to explain.
Dear Professor Gelman,
On one hand, “start by graphing the data” is sound advice. On the other hand, so is “start by learning how to count.” I think you are suffering from a set of experiences in which you have been overexposed to analyses where people have not graphed their data. In reality, though, not-graphing-data is not a problem in the wild. ask anyone who works in finance, in the pharmaceutical industry, in marketing. In fact there is a huge industry today called “data science” dedicated to the premise that graphing data without paying attention to rigorous inference is quite sufficient—what really matters is beautiful pastel colors and charts that look like floating violins. Perhaps you could be a little more balanced in your blog—for example, if you are unwilling to give up on the infantile advice to plot data (omg) you could at least say things like “first plot the data to ground your analysis in common sense, but then don’t forget to actually do some rigorous statistics.” There is a generation of readers of your blog who think graphing is the whole story.
Sincerely,
Someone who doesn’t want to be plotted on a crucifix with beautiful pastel blood pouring from the gash in his ribcage.
Worried:
I completely agree with your advice to “start by learning how to count.” In my view, measurement is the most important thing in statistics that’s not in the textbooks. Over the years, we’ve talked about lots of examples where researchers have no control over their data, where they can’t even keep track of what data they have or how these data were recorded. Garbage in, garbage out, and the first step is to look at what is coming in.
In the above post I was responding to a specific question from someone. Based on his email, I assumed that he had already “learned how to count.” He was asking questions about a particular statistical model, so it made sense to me to emphasize that (1) there’s value in looking directly at the data before getting swept up in the details of modeling, (2) there’s value in looking at the data after fitting the model to see what more can be learned, and (3) modeling is a process of learning from the data, it’s not just about fitting one model that’s been chosen ahead of time.
Regarding your particular points:
1. You write, “In reality, though, not-graphing-data is not a problem in the wild.”
In reality, “the wild” is a big place, and I’ve seen lots of examples of people not graphing their data, in settings where I think that graphing could make a difference in their work.
2. You call my advice “infantile.” I don’t think this adjective makes any sense here.
3. You talk about “rigorous statistics.” Graphical methods can be rigorous. What’s important is not just graphing, but using graphing and modeling together.
4. Regarding being “more balanced”: You should start your own blog! It would be great to have a blog out there focused on the importance of data quality. You could all it, “Learning how to count.” It’s an important topic. I’m just one person and I can’t cover every statistical topic in depth here. Data quality doesn’t get enough attention in statistics.
Andrew, what is your view on the issue (that you have discussed yourself a lot) that model-based theory normally assumes that models and methods have not been chosen conditionally on how the data look like? Obviously, if you plot the data first and your modelling is inspired by what you see there, this assumption is already out of the window and we’re in forking paths territory.
Now for sure I’m not saying that we should not start by looking at the data. In fact my take would probably be something like violation of assumptions due to looking at the data first and doing something conditionally on that is normally not worse and often better than not even seeing that the model is bad (although it may be worse if the model chosen without looking at the data is fine).
In any case I believe this is a thorny issue, and when I was younger some professors indeed said don’t plot the data first as this will invalidate analyses (at least if there are consequences from looking at the data). So I’d appreciate a lot if you could comment on it.
Christian, you articulated quite nicely my concern over graphing prior to analyzing. In an ideal world, we wouldn’t let the graph drive our modeling decisions, but in practice, I imagine it’s quite hard to resist tossing out that outlier point showing up on the plot.
Once we have a graph, we start exploring it and, then, well, everything just turns into exploratory data analysis. That in of itself may not be a bad thing, but I just feel that every single study is exploratory, because so much of the learning from data happens before we model. It would seem not graphing would help stop that from happening.
I find the idea that you should “analyze” the data first because graphing it imposes too many restrictions on what we learn from the data – ludicrous. The points about producing quality graphs and not just ones that look nice is certainly valid, as are the cautions that graphing the data is not innocent, and not without its own assumptions.
But a few of these comments have lost perspective. Given a choice of looking at the data before modeling or modeling first before looking at the data – I can’t see any sense in the latter rather than the former. This doesn’t mean the analyst is freed from making assumptions or from forking paths or from data measurement issues. These issues are all real. But given the myriad modeling choices, I can’t see any sense in building a model first without looking at the data in any way. What model would you build? How would you detect measurement issues in the data? Is the suggestion that p values would be a good place to start?
When the number of potential factors becomes large, then visualizing the data certainly becomes more difficult – perhaps to the point that some kind of “analysis” is needed before any visualization is possible. So, you may need to do some kind of predictor screening before it is feasible to visualize anything. But I view this as one of the major difficulties with such data sets – absent early visualization, we are somewhat in the dark about what kind of analysis to do. In my view, this does not improve the resulting analysis – it impedes it.
Nothing I am saying defends creating pretty or eye-catching graphs at the expense of rigorous analysis, and I don’t think Andrew is defending that either. But the fact that some visualizations are counterproductive is hardly an excuse for forgoing visualization in favor of blind analysis (pun intended).
Christian:
I recommend always looking at the data first. If there is concern about forking paths in model selection, you can always gather more data. If you can’t gather more data (for example, if you’re analyzing historical data on business cycles), then you won’t be coming at the data from a completely fresh perspective anyway.
Turn:
Yes, everything is exploratory data analysis. That’s ok! Again, if you want to preregister an analysis and gather new data, that’s fine too. I just think that the vast majority of science and engineering is exploratory, and that’s ok. We’re exploring and learning and trying new things.
At some point we want to go beyond exploration—for example, if you’re building a bridge, you don’t want it to collapse under traffic. For that, you want some strong theory and experimental evidence. I don’t see how graphing your experimental data would be a bad idea there either!
Andrew: I agree with the recommendation, however I wonder whether you are (or aren’t) concerned about bias introduced by informal data dependent selection of models and methods. Obviously if we are concerned, this doesn’t imply that it shouldn’t be done. It may mean that results should be interpreted with more care and less confidence. Also, quite often indeed it’s at least hard to gather more data as this comes with cost and effort, and the time and money may not be available.
There are two key steps of the scientific method. First is abduction, where you guess models based on the data. The second is deduction, where you deduce the consequences of the models and compare those predictions to new data.
Both steps are necessary. This is a great example why: https://en.wikipedia.org/wiki/750_GeV_diphoton_excess
That shows dozens (even hundreds) of plausible explanations can be generated per month to explain something that apparently doesn’t even exist.
Or…how about graphing half the data?
@John N-G Is that what you recommend, and practice? Just for visualisation or for the modelling that comes afterwards, too? In which case you use the other half only for validation I guess? Well if you have enough data, fair enough.
As a very much applied statistician working on a diverse variety of projects in many different subject areas, not plotting my data first would be a huge waste of time. Many times I have found errors in the data entry process, simply by plotting my data first. I would hate to think that I wasted hours on some data that I hadn’t checked over first, all for the reason that it was somehow biasing my analysis by viewing it prior to modeling it.
Here’s a real world example: You are working with some lab experiment data where they grow bacterial colonies on blood agar plates. They count the number of colonies on each plate. The tech emails the dataset and the outcome variable, counts, looks something like this:
counts <- rnbinom(n=1000, mu=60, size=0.8)
counts 250, 250, counts)
Now, you decide that you don’t want to bias your analysis by looking at the data, so your proceed to do:
summary(counts)
Nothing to see here.
You proceed to model with negbinomial or Poisson. Model doesn’t check well.
Or, you decide to plot a simple histogram:
hist(counts, breaks=100)
Whoa, something is def weird, so you message the tech:
me: Hey, why are there so many counts at 250?
tech: Well they’re TNTC.
me: What is TNTC?
tech: too numerous to count
me: Too numerous to count? So you just stop counting at 250? Why not write TNTC?
tech: yep. If there are more than 250 we stop counting. Well, in the past if it was ever TNTC we just put the max number and that way we could still run the t-test.
me: Gotcha… and why do you stop counting?
tech: Well, we figure there is a lot of error when there becomes that many colonies on the plate. Also, sometimes the colonies run together and grow on top of each other, and it is hard to decipher them.
And now you know. You didn’t bias your analysis, but rather you have information on the data generation process. Unless you are the person working directly in the lab and gathering the data, then there is really no way for you to know about ‘TNTC’ unless you are forewarned (which may or may not happen in the real world).
I wonder if the same people who would advise not to plot your data first are also the same people who think that posterior predictive checks are a bad idea?
somehow the ifelse statement was lost: counts = ifelse(counts > 250, 250, counts)
“After you’ve fit some models, you can graph the data along with the fitted models and look for discrepancies” — seems automatable, no? Presumably, discrepancies = heteroskedasticity? Options = nuke the point(s), expand the model?