Brett Cooper writes:
I read your book Regression and Other Stories. As a beginner and community college student, I wonder if I may be able to ask you a couple of simple and clarifying questions.
1. Multiple Regression and Collinearity
I created 2 linear regression models using variables from a simple dataset. The first model is a simple linear regression model with a regressor X1 and a specific response variable Y. The other model is a multiple linear regression model which includes X1, X2, X3 and Y. The statistical summary for the simple linear regression model assigns a positive and statistically significant coefficient beta1 to X1 indicating a positive linear association between X1 and Y. However, the statistical summary for the multiple regression model shows that the coefficient beta1 for X1 has changed sign, changed magnitude and is not statistically significant anymore.
Did that happen because of the undesirable but often unavoidable effect of collinearity between X1 and one or both of the other two predictors X2 and X3?Based on my understanding, it is to be expected that, even with zero multicollinearity, the regression coefficient associated with certain predictor X change, in magnitude or even sign, when the predictors X are considered together in a multiple linear regression model instead of separately in relation to the response variable Y. Is that correct?
However, is it “normal” for the coefficients’ sign and for the p-value for the same predictor X to change when switching to a different model? A change of coefficient sign for X would indicate an opposite behavior between X and Y in going from a simple to a multiple regression model…— As a rule of thumb, before creating a multiple regression model involving Y,X1,X2,X3, would it recommended and useful to first create the simple regression models, i.e. Y =beta1*X1+beta0, Y =beta2*X2+beta0, Y=beta3*X3+beta0, and compute their regression coefficients? Or should we jump straight to the multivariate model Y= beta1*X1+beta2*X2+beta3*X3 and evaluate the regression coefficients at that point?
2. Multicollinearity
— Multicollinearity affects the interpretability (impacts the accuracy of the regression coefficients) of our model but not its predictive power. Multicollinearity can have different sources: it can originate from the data itself but also from the structure of the model. For example, model Y = beta1*X1 + beta2* X1^2 + beta3 * (X1*X2) has interdependent terms. Surprisingly, multicollinearity would NOT be present between X1 and the term X1^2 even if they are quadratically dependent…Is that correct? What about the interaction term (X1*X3) and the term X1? Or would multicollinearity be present and only be reduced if we mean center the variables X1, X2, X3?
— Correlation means linear dependence. Is multicollinearity only caused by the presence of linear dependence or do other types of dependence (curvilinear, etc.) between predictors also cause collinearity in the model?
3. Variable Transformation (feature scaling)
–Certain statistical models require their input predictors to be scaled before they can be used to build the model itself so the variables can all be on equal footing.
Some models, like decision trees, don’t require scaling at all. Scaling variables (linear or nonlinear scaling) is generally useful when the involved variables, the Xs and the Y, have very different ranges. My understanding is that predictor variables with large ranges would automatically receive large regression coefficients even if their relative importance is lower in comparison to other predictors. Is that correct and true for most models?— In some cases, scaling seems optional and only improves the interpretability of the association between Y and the Xs (computed correlation may be tiny only and we can increase it by scaling the variables). That said, is the scaling of the predictors X and/or response variable Y necessary and critical for the creation of an accurate and correct simple or multiple linear regression model? Or does scaling only help with the interpretability of the regression coefficients?
My reply:
1a. When you add predictors to a model, you can expect the coefficients of the original predictors to change. Once they can change, yes, they can change sign: there’s nothing special about zero. As for statistical significance and p-values: sure, they can change too. One way to see this is to imagine N = 1 million: then even small changes in the coefficients will correspond to huge changes in p-values.
1b. Yes, when you fit a big model, I recommend fitting a series of little models to build up to it, and you can look at how the predictions of interest change as the model builds up.
2. I’m not completely sure about your questions on multicollinearity. To put it another way: the answers to these questions are not obvious to me, and I recommend figuring these out by just simulating them in R.
3. I think that scaling is important for the interpretation of parameters (see here) and if you’re going to use Bayesian priors (see here). There’s also nonlinear scaling (logs, etc.) or combining predictors, which will change your model entirely.
For answering questions on coefficients changing sign (1a above) I like the following example (I wish I could remember where I first saw this so I could give proper credit):
Consider a dataset based on choosing people at random and counting the coins that they have in their pockets (ok, somewhat dated example, but pretend people still carry physical money). The y-variable is the total value of all the coins, the x1-variable is the total number of coins, and the x2-variable is the total number of pennies and nickels (or the lowest 2 denominations of coins for non-US cases). Clearly the slopes will be positive when regressing y on either x1 or x2 (but not both), but when both x1 and x2 are in the model, what is the effect of x2? One way to interpret the individual slopes is the change in y for a 1 unit change in x2, while holding the other predictors constant. So what is the effect of increasing x2 by 1 coin, while holding the total number of coins (x1) constant? This could only be done by replacing a higher valued coin with a lower valued coin which is clearly a negative slope on y. This example makes the sign switching more intuitive and obvious for me.
How about following that up with an airfare example? Regressing average airfares (by route) on the number of miles shows a positive relationship, airfares on market share of the largest carrier looks negative (which is highly counterintuitive). But when both distance and market share are in the model, then market share has a strong positive relationship with average fares. The reason for the change is that market shares for the largest carrier are inversely related with distance (many short routes from hub airports are dominated by a single carrier – very few long distance routes are).
I much prefer using real and complex data, but starting with your simple intuitive example and the moving to a more realistic and important context makes some sense to me.
Dale,
I agree, thanks for the real data example.
I guess my intuitions about airfares are different. I would have thought that the largest carrier achieved that status, in part, by having lower airfares, so the negative relationship makes sense. That it switches when you add distance is what requires some explanation, making the example less clear to me.
Here’s a real data example that I use when teaching multiple regression. Air pollution in a city is positively related to the number of factories in the city, and also positively related to the population size. To me this makes sense– more factories -> pollution, and more people -> more pollution. But when you include both factories and population, the effect of population becomes negative. This also makes sense: for a given number of factories, the more people you have the more concern about pollution you have and the greater ability to enact air quality legislation. (The example is from Sokal and Rohlf’s Biometry.) It allows students to see very quickly that the direction of a relationship is not “fixed” in some way, but depends on what other factors are included in the model. But intuitions may vary, so having multiple avaialable examples is a good thing.
I guess our intutions differ. Your example (air pollution, factories and population) seems questionable at best. The airfare situation, on the other hand, seems completely sensible to me. Anybody who has lived in a smaller community served out of a hub airport with one dominant carrier knows very well that the prices are high, not low. In fact, it usually costs more for the flight to the hub airport (short flight) than from the hub to another large city (long flight). To each their own.
Whether the airline or pollution example, I’m still not clear on how to practically implement Andrew’s advice of “fitting a series of little models to build up to it, and…look at how the predictions of interest change as the model builds up.” Presumably we know a priori that airfares vary with distance and market share and pollution varies with number of factories and people – that’s why we collected those data in the first place.
Hence, what do we gain by first interpreting the regression coefficients and predictions from the simple model and then again from the two variable (more complex) model? Or could it be simply to show that our intuition about simple relationships are wrong, once we start to explore other factors?
I keep harping on this issue because I’ve seen a number of papers that just choose which model to interpret – simple vs complex – based on what best fits the original hypothesis, without explaining the nuances like Dale and Gregory do in their explanations.
To me, fitting the simpler models and building up is to shed light on how the additional factors reveal more about the processes at work. For the airfares, it becomes clear that distance is a prime influence on airfares, and that market share matters but without distance included, the sign of its effect is wrong. Further, if you run these models over time (run the same model for each time period rather than including time as a factor – a suggestions Andrew made a little while ago), you find some interesting things about the size and significance of the market share variable (in particular, it was a large effect until the large airline mergers happened and has become fairly small since).
I don’t think it is legitimate to pick and choose which model to report – if an effect changes markedly when other factors are included, then I think the simpler model is highly suspect. Of course, the more complex model may be suspect as well, but there is little to recommend using a model that you have already seen does not capture the correct direction of an effect. I don’t believe there is any way to completely negate the critique that a model may omit important factors that would alter the relationships that had been “discovered.” Omitted variables will always be an issue, especially due to the limits of data availability. But the process of building up from simple to more complex models is one way to get a sense of how fragile or robust your model is.
Thanks for this great teaching example!
Here is how I simulated it:
“`
coins <- c(2, 1, .5, .2, .1, .05, .02, .01) # Euro coins
wallet <- sapply(rpois(200, 5),
function(s) sample(coins, s, replace = TRUE))
dat <- data.frame(
value = sapply(wallet, sum),
ncoins = lengths(wallet),
nsmall = sapply(wallet, function(coin) sum(coin < .05))
)
plot(dat)
lm(value ~ ncoins, dat)
lm(value ~ nsmall, dat)
lm(value ~ ncoins + nsmall, dat)
“`
I hope that it is ok to leave multiple comments rather than one really long comment.
For 1b above, I like doing both the single variable and the multivariable regressions because they answer different questions which can all be of interest. Even better is create the causal diagram from the science and let that help you decide which models answer which questions and which are the most interesting.
You also need to be careful when doing this. In the NHST world I have been asked to fit the single predictor models, then use the p-values from those to decide which predictors to include in the multiple regression.
My response is to show a simple simulation where I simulate x1 and x2 with a moderate, but not super strong, positive correlation, then simulate y from a model based on x1 – x2 (plus random error). With a reasonable sample size it is easy to get the case where both single predictor models give non-significant p-values (even at alpha = 0.1 or 0.2), but the multiple regression model correctly shows very significant results, but would not be seen if both x1 and x2 were “screened” out due to the simple models.
Going the other way it is easy to simulate 50 predictor variables (not correlated) and a response variable that is just random noise (not dependent on the predictors, any observed relationship is pure random chance). Now choose a subset of the predictors based on their “significance” in predicting the response (either 1 large multiple regression with all 50 predictors, or 50 1-predictor models), some of the predictors will give “significant” results due to chance (type 1 error). Now fit a regression with just the “significant” predictors from the screening and you will see a model that looks really good, but is completely meaningless (in frequentist stats, the assumptions for the p-values are majorly violated, resulting in meaningless results).
Both these examples are good arguments against NHST, but could also cause problems with a Bayesian analysis that is not thought through (choosing which variables to include in a “full” model based on credible intervals including 0 or not). Using strong regularizing priors would lessen the problems in the second example, but increase the problem in the first. Weaker priors, or priors not centered at 0, would lessen the problem for the first example, but increase it in the second.
The problem here is not frequentist vs. Bayesian, but rather not using what is known about the science and just relying on the data and a screening procedure.
Can you give a practical example of where you interpreted simple and multiple regressions in the same paper? I found Andrew’s advice of “look at how the predictions of interest change as the model builds up” a little too vague to be helpful. When writing a paper, what would one report, exactly? In my understanding, the simple regression would show you the Y~X relationship in the presence of a lot of noise because the error term subsumes any variation due to other independent variables. I’m not clear on how that would be useful, compared to, say solving for Y = beta_1*X_1 + beta_2*X_2 and setting X_2 to zero, or another value of interest.
In cases where I’ve seen both simple and multiple regressions reported in the same paper, I’ve found the authors tend to interpret the coefficient that best confirms their hypothesis, brushing the other aside.
Here is a paper that I was involved in: https://pubmed.ncbi.nlm.nih.gov/35960508/
Unfortunately the full article is behind a pay wall (though I found a full version by going through a university library) and the most interesting parts were compressed to a single sentence in the abstract.
It does not include the single predictor models, but it does include the full model with all the predictors, then several other parsimonious models with 2 – 4 predictors.
When my co-authors first came to me they wanted to find the single “best” model and best tool for predicting improvement in low back pain. There are several tools of interest (mainly scores from surveys given to patients at their first physical therapy visit) and the idea was to find the single best tool to use (or small combination of tools). When I showed them that while there is a “best” model, that there were several other models that were almost as good, they were excited and we changed the whole focus of the paper to show that while 1 tool was in all the preferred models, the second tool to use could be decided by the therapist or practice group without making any real difference (and many have their favorite, so this meant people could use their favorite and not argue over which is “best” when the differences are not worth arguing over).
Thanks for the reply! I should read the paper for specifics, but I’m sitting at my kid’s soccer practice. Apologies if my answer is in the paper.
But from what you wrote, it sounds like something different than what Andrew advocates. It sounds like you’re dealing with a model selection problem, and deciding on which variables – tools, in your case – are “important.” But I’m questioning how to derive inference from different coefficients of the same variable in the same paper. What if the tool had a positive effect in the single variable model, but a negative effect in a model with another tool included? Why would you report the effect from the single variable model? (And I’m aware that one convention is to take the model weighted average of the coefficient).
On point 3 from Brett Cooper: The questions asked go like this: ‘…is it necessary and critical… for a correct model… or does it just help with interpretability?’ I would first say that all models are wrong (although some are useful), so no, scaling will never lead to a ‘correct’ model, i.e. a model that describes the real world outcomes 100% correctly. However, as has been pointed out, some models (e.g. principal component analysis, LASSO) depend on the scaling of the input variables. Let me give an example: The variance of a random variable depends on the scale on which the variable is measured. The variance of length of the 100 nails from a box of nails changes of if measure and calculate in inches or miles. The variance will be ‘normal’ if you use inches, but basically zero if you measure it in miles. So if a model parameter depends on the variance and the effect of the variance is not cancelled out in some way (in OLS the scales cancel each other out), the parameter changes! Sometimes we therefore have to remove these scaling effects ourselves. This is necessary for replicability and comparability.
On the question of interpretability: I guess it depends. I find it quite difficult to understand normalised data. So if normalised X1 has some effect on normalised X2, I understand the sign of the parameter, but I have a hard time understanding the magnitude. Maybe it’s just me, and training can fix all that? I think one would have to rescale the parameters to the original scale (after calculating them using normalised data) for interpretable units of the parameter.
A couple comments on point 2 above.
First, x1 and x1^2 will only be uncorrelated if x1 is centered at 0. If x1 has only positive values then x1 and x1^2 will be correlated (and the correlation will be stronger the further the x1 values are from 0). This is one argument for centering variables.
But it can get more complicated than the original questioner asked about. If x1 and x3 are correlated, then x1^2 and x1*x3 will also be correlated, which can cause difficulties for interpreting the model, is the relationship best determined by a non-linear term in x1, or an interaction between x1 and x3, and including both, without some form of regularization, will give such wide intervals to make any interpretation questionable.
Interesting discussion. FYI, I published a review paper on interaction effects a while back (2011), and there was a summary of it in R-bloggers in 2018.
https://www.tandfonline.com/doi/abs/10.1080/10691898.2011.11889620
https://www.r-bloggers.com/2018/05/regression-with-interaction-terms-how-centering-predictors-influences-main-effects/
Regarding coefficents changing sign: We have to be very careful and should be skeptical when signs of existing variable(s) in regressions change upon the inclusion of a new variable correlated with an existing variable. Yes, it can be the legitimate result of separating out true effects that were previously confounded. But, if there is an omitted variable correlated with both variables (existing & new), it can push the signs of the variables in opposite directions completely illegitimately. The result is a type 1 error.
I published a paper on this phenomenon back in 2018, and an update in 2022.
https://onlinelibrary.wiley.com/doi/full/10.1002/smj.2783
https://link.springer.com/article/10.1057/s41267-022-00531-9
Sadly, I am getting the impression that very few of the contributors to the current topic 1) have a good understanding of regression and/or 2) are familiar with the extensive decades-long litereature on data-driven model selection. Some of the comments sugggest a fundatmental lack of understanding of the difference between a bivariate and a multiple regression model for the same dependent variable. This is elementary stuff. Data-driven model selection involves more than simple seat-of-the-pants approaches, especially when some form of NHST is being used as a decision criterion.
re “coefficients changing signs”
I have seen this frequently in health care risk adjustment (regression) models, which tend to have a lot of predictors. These are often “perverse predictors”, because the sign is the opposite of what you would logically expect from a clinical point of view.
What I believe happens, and I have proven in a few cases, is that there are some other powerful main effects that have an interaction, but that interaction is not specified in the model. The model thus over-predicts the impact of the two (or more) variables in combination, which forces the underlying math to select the best available predictor to serve a proxy for the effect of the missing interaction term … as a relief valve, in a sense.
@Brett Cooper w
1a: Yes if new variables X2, X3 are included the magnitude, sign as well as statistical significance can change. The extent of this change depends on two things
i) Correlation between X1 and other included new variables
ii) The other variables X2, X3 are really predictors of Y. If they are useless for Y, (and uncorrelated with X1), coefficient of X1 in larger model will be only has a negligible change.
I thinks if your expect that x1, X2, X3 are really determinants of Y, there is no point in estimating simple regressions.
2. Multicollinearity refers to linear association between predictors. Non linear yet perfect correlations can be used in the model without any problem e.g. X and X^2 have perfect non linear correlation, so theoretically speaking they can be used without fear of multicollinearity.
— Multicollinearity affects the interpretability (impacts the accuracy of the regression coefficients) of our model but not its predictive power.
yes
3. Variable Transformation (feature scaling)
Interpretability is certainly a reason of transformation. Other reasons are to be able to satisfy model assumptions e.g. constant variance, normality more closely.