This is my talk for a seminar organized by Joe Suzuki at Osaka University on Tues 10 Sep 2024, 8:50-10:20am Japan time / 19:50-21:20 NY time:
Effective Number of Parameters in a Statistical Model
Andrew Gelman, Department of Statistics, Columbia University
Degrees-of-freedom adjustment for estimated parameters is a general idea in small-sample hypothesis testing, uncertainty estimation, and assessment of prediction accuracy. The effective number of parameters gets interesting In the presence of nonlinearity, constraints, boundary conditions, hierarchical models, informative priors, discrete parameters, and other complicating factors. Many open questions remain, including: (a) defining the effective number of parameters, (b) measuring how the effective number of parameters can depend on data and vary across parameter space, and (c) understanding how the effective number of parameters changes as sample size increases. We discuss using examples from demographics, imaging, pharmacology, political science, and other application areas.
It will be a remote talk—I won’t be flying to Japan. Here’s a link to the session.
It feels kinda weird to be scheduling a talk nearly a year in advance, but since I had to give a title and abstract anyway, I thought I’d share it with you. My talk will be part of a lecture series they are organizing for graduate students at Osaka, “centered around WAIC/WBIC and its mathematical complexities, covering topics such as Stan usage, regularity conditions, WAIC, WBIC, cross-validation, SBIC, and learning coefficients.” I’m not gonna touch the whole BIC thing.
Now that we have loo, I don’t see any direct use for “effective number of parameters” in applied statistics, but the concept still seems important for understanding fitted models, in vaguely the same way that R-squared is useful for understanding, even though it does not answer any direct question of interest. I thought it could be fun to give a talk on all the things that confuse me about effective number of parameters, because I think it’s a concept that we often take for granted without fully thinking through.
Agreeing to give the talk could also motivate me to write a paper on the topic, which I’d like to do, given that it’s been bugging me for about 35 years now.
I agree that “effective number of parameters” is not something I see a direct use for. I disagree that it’s indirectly useful. Or at least if it’s indirectly useful it’s only to people who have developed an intuition around classical statistical hypothesis testing. Since I have no such intuition, if you told me I have 120 parameters in my model but it’s “effectively” 23 it would be meaningless to me. I wouldn’t know what to make of that.
I could understand posterior entropy for example, or the ratio of parameters to data points… Or the marginal information gain from a single data point… But I don’t know what to think of “effective parameters”
I guess maybe I should watch the talk but it’s a year out!
I’d rather see out-of-sample predictive skill per computational cost (ram, cpu-time, etc). Both roughly correlate with number of parameters, which is what this effective number is trying to get at I think.
Usually when we are worried about that the individual parameters don’t really mean anything though. I mean they do mean *something*, but it is conditional on the other 100 or whatever parameters and structure of your arbitrary model.
For meaningful/interpretable parameters we still do want as few as possible.
Do you intend to discuss the effective number of parameters in statistical models used for inference vs those used for prediction? Is there a difference? It seems to me that many machine learning modelers worry a lot less about the number of parameters and are focused on prediction rather than inference. But I believe there is (much) more to be said about that. Is that part of what you are planning?
“…inference vs […] prediction? Is there a difference?” There is something in it that I cannot wrap my head around: the distinction between inference and prediction. In my book, causality is the strongest form of prediction. I can of course see the differences between these ‘paradigms’, which are about the interpretability of parameters and their distribution. But in essence they are the same: we fit a mathematical model to data using some loss function (e.g. RSS). What determines how close the model can get to causality is the data, not the model (GIGO).
So I don’t see why ‘inference’ and ‘prediction’ are often treated as dichotomous. (I don’t mean to imply that Dale Lehman treats them as dichotomous, but it was something I needed to get off my chest).
I’ve always thought of inference as a type of fact that we want to elucidate from some data/model combination, and these could be predictive, causal or descriptive (as a useful, if not fully accurate, classification). So, predictive inferences can be contrasted with causal inferences, although in some cases they might overlap.
I agree with this perspective. As an extreme example of how causal and predictive inferences can differ, consider the “gold standard” for causal inference – the randomized trial. A two-arm individually randomized trial where we’re interested in mean differences can be analyzed using something as simple as a two-sample t-test. This is rock-solid from a causal inference perspective, but is terrible as a prediction model.
That’s an interesting perspective and one I mostly agree with. However, the practice of many data scientists concerned with prediction is to not worry about causation or inference regarding any specific variable(s) – indeed, some models have so many parameters as to be largely uninterpretable. I know this topic has received much attention and at least dates back to Brieman’s two worlds. While my training and inclination is to want models that can be interpreted and at least suggest causative factors, the predictive success of models without these features is often impressive. So, I think there is till much to be said and it sounds to me like Andrew’s subject is related to such things.
Say you want the volume of a box. We know that is v = l*w*h. First, the causality isn’t clear in this case. You could just as well write h = v/(l*w), etc. But what if we did not know that equation and wanted to estimate the volume of some boxes from whatever measurements are available.
You could get data on all kinds of boxes with known volume. Stuff like mass, color, material, year of construction, location, and so on. The features and volume can even be measured with some error. Then fit an arbitrary model like:
v = B1*mass + B2*color + B3*material + …
All those features are correlated somewhat with the volume of boxes, so your model will be able to predict the volume better than if you knew nothing at all. But this process will be very inefficient (relative to using l*w*h*), and none of those coefficients mean anything.
Then say you get some new data, on the length of the boxes. So you add that as a new term to your model. This will then change all the other coefficients. Attempting to do inference on the parameters/coefficients of such arbitrary models is very common, and very silly.
I’ve seen you make this argument before – but I don’t find it convincing although I do agree with it as a general caution. Sure, we know that v = l*w*d and if we are missing one of these variables and later add it to our model, the coefficients for the others will change. It is always a valid caution and one that I am keenly aware of that omitted variables create problems, perhaps serious, for inferring anything about the included variables in a model. It is also a reason why I think an important check on a model (to use Andrew’s un-favorite term, a robustness check) is to see if the model coefficients change markedly when new variables are added.
But I don’t think that proves a general point that it is silly to do inference on the parameters/coefficients of a model. We are often dealing with models such as: price of house = f(size of house, size of lot, features, age of house, etc.) or cost of a particular treatment = f( age of patient, some measure of disease severity, where treatment takes place, type of insurance, etc.). Such models are frequently encountered, and all have omitted variables. But does that make them all “silly?” Perhaps your qualifier (“arbitrary models”) is what makes them silly. I’m all for deriving models from some mechanism – which presumably makes then not “arbitrary.” In many fields, the potential mechanisms is large and there may be great variation in these. Are you saying that all such modeling efforts are “very silly?” If so, I find it hard to agree with you, although I think it is valid to caution against believing any particular model too strongly. Shouldn’t we evaluate such models on whether they suffer from omitted variables that theory would suggest matter, and whose inclusion might dramatically change the inferences derived from the model? Saying all such models are “very silly” appears to say that we should never build models unless we have every relevant variable included or unless we know the true model (as with the volume formula). In most fields, I don’t think these conditions hold.
I think we are in the realm of “all models are wrong, some models are useful.”
I also think this is connected to the comment I made yesterday on another thread about “science” covering a lot of ground and issues that are important in one area of science are not important in others. And that goes double, at least, if we have a very broad definition of ‘science’ to include, say, forecasting what my house would be worth if I tried to sell it next month versus deciding to wait until a year from now. Obviously it is not possible to predict those numbers from some sort of first principles mathematical of physical calculation. I think even Anon would agree I can at least get a ballpark estimate by using one or more statistical approaches, e.g. for forecasting the selling price next month I could get data on sales of nearby homes over the past year and fit a model to try to adjust for trend, house size, lot size, year built, interest rate, month-of-year or season, and so on. The model might or might not give a more accurate answer than I could get just be asking a realtor, but I think even Anon would agree that fitting such a model would not be ‘silly.’
I strongly agree with what I _think_ is Anon’s underlying point, which is that many scientific theories are not statistical in nature and perhaps not even amenable to statistical insights. Newton did not work out his theory of gravity by assuming F = -GM*m/r^k and then collecting data so he could fit them to try to estimate k. Doing so might or might not have been ‘silly’ but I think we can all agree it would have been a waste of Newton’s time.
On the other hand, I think ewe would also all agree that it would be ‘silly’ to try to forecast next year’s value of my house by trying to work it out from causal principles.
That is fine for predictions, it should do better than no model at all. It isn’t the model that is silly. It is trying to interpret the coefficients. The parameter estimate for the effect of size of house, etc doesn’t mean anything. The size is correlated with the price just like the material a box is made out of correlates with its size, same thing.
Even when used for prediction, the omitted variables can also ruin your business model:
https://www.cnbc.com/2021/11/03/zillow-stock-plunges-24percent-after-company-exits-home-buying-business.html
But their model was probably still better than nothing.
I think I am in substantial disagreement with anon here. I’ve constructed many such silly models of housing prices, for example. The parameter estimates for the size variable are not meaningless. They appear to reasonably represent what construction costs for houses are in different locations and at different points in time. They do vary depending on what other variables are in the model, and that is indeed one way I evaluate each model. When the coefficient for house size varies greatly when other variables are included, I find that model wrong and not very useful. When the coefficient stabilizes, I find it wrong but useful.
As I find common with anon’s contributions, they seem overstated and too extreme. It does seem that anon really believes their stated position – in this case, perhaps house price models are only good for prediction if they are useful at all. But I have found them to be fairly reasonable descriptions of how various factors influence house prices. As a model for the price of any particular house, the prediction interval is too wide to be useful. As estimates of how a number of factors (garage size, age of house, finished basement, for examples) I find the estimates often reasonable and useful. I would make similar statements about a number of other examples: earnings as a function of college and major choice, airfares as functions of features related to different routes, etc.
The fact that such models can go wrong (as with the Zillow example anon cites) is certainly informative. But the lesson I learn is not that it is silly to pretend that such models are informative. It is that all models are fragile in a number of ways. No model will capture factors that do not exist in the data is was trained on.
By using “they”, It seems you are talking about the entire model (all parameters together). Ie you keep talking about using the model to predict something. This is correct.
In machine learning they do use “inference” to refer to making predictions, which is a confusing overloading of the term imo. This is different than calculating CIs for neural network weights or similar and seeing if they are significantly different from zero or whatever. No one does that.
What is the usecase for the house size coefficient? To me it is like a neural network weight.
The use case is to understand the factors that influence house prices. Size of house, age of house, particular features, neighborhoods, all contribute to house prices and I believe we can quantify these factors. Of course, whatever model we build will be wrong – and some will be worse than others. But I believe it is possible to have some confidence in those parameter estimates (and their associated uncertainties) if we include the factors we believe to be important and if the estimates don’t change markedly when adding additional factors to the model. Even a “good” model is unlikely to be of much use for individual predictions, such as how to price your home. But if we want to understand average effects or average effects within various subgroups, then the model may be useful (size of house affects house prices differently in different locations – and construction costs also vary in different locations).
Maybe this is obvious but: whether the individual coefficients are directly interpretable outside the rest of the model is dependent on the model and the context of the model. Sometimes the answer will be yes and sometimes it will be no.
In the value-of-a-house example, suppose I’m thinking of building a 300-square-foot addition onto my house and I want to know how much this will increase the value of my house. The addition would not include a new bathroom or bedroom or anything else that is included in the model (assuming the model includes ‘number of bathrooms’ and ‘number of bedrooms’ as input variables); let’s say I’m extending the living room or something. Can I get a decent estimate of the increased value of the house by multiplying the ‘square footage’ coefficient by 300? If I can then interpreting the coefficient on its own is meaningful.
That said, Anon makes a point that I agree with and that many people don’t really understand no matter how many times it is pointed out on this blog or elsewhere: for most models and most parameterizations, all of the coefficient values are tied together in a way that makes it nonsensical to try to interpret them separately. In general, larger houses have more bedrooms and more bathrooms and more windows etc. etc., so when you fit a model that includes a range of houses all of these coefficients end up co-varying in such a way that you cannot in fact accurately use the ‘square footage’ coefficient in the manner described in the previous paragraph. Maybe it will work out fine if your 300-square-foot addition happens to make your revised house match the average house that is 300 square feet larger than your current house, but if the average house that is 300 square feet larger than your current house has an additional bathroom, but you aren’t adding a bathroom, then it’s not going to work very well…although, on the other hand, it may work well enough to be useful.
I haven’t said anything here that we don’t all already know — that is, Dale and me and Anon and presumably anyone else who has read this far. It’s my suspicion that Dale and Anon don’t actually disagree on anything substantive but you’re talking past each other a bit.
Phil
On your point about the jointly determined coefficients – absolutely. But we can deal with multicollinearity (up to a point). We can detect its presence and we can see how those related coefficients vary as one or more of them are dropped from the model. In any case, I don’t agree with anon’s conclusion that the fact that all coefficients in the model are related means none of the coefficients are worth looking at.
On your example of the house addition: I would say that would be a misuse of the model estimate for the sq ft variable – if the data the model was developed from is for existing houses, some of which may have been expanded, but mostly have not. I’d far prefer a data set of house additions. The better use case might be a developer who is considering a new development and thinking about house sizes and other features that will impact its sales price.
Apparently we have moved on to the question of ‘what parameters can be interpreted and if so, what do we need to consider’. Again, I would tie the question back to the data. Reiterating a previous point: We would not trust a model of general house prices to model the value-added of an extension. But this is not the fault of the model or the parameters. It is the “fault” of the data. In this respect, I echo previous comments. I suggest that if we want to interpret parameters, we should interpret them in the light of the data generation process of the data that trained the model. And if we are not comfortable interpreting a single coefficient (because you are unlikely to find a house with one bedroom, three kitchens and a dozen bathrooms), I suggest we compare cases. Simulate a realistic scenario for which the model has been trained, and compare it to a closely related scenario but with one change (e.g. similar house, different neighbourhood). For linear models, this would be equivalent to directly interpreting the parameters in the ‘standard way’, but for models with parameters that are difficult to understand, this approach seems to be appropriate.
Assume you have cells that undergo binary division, ie one parent results in two daughter cells. So starting with N0 cells, after d divisions there are:
N = N0*2^d cells
Then we can measure number of cells over time, and model the rate of division, where d = r*t:
N = N0*2^(r*t)
Simple enough, but then we collect data and see actually the population grows slower because some of the cells die out. For simplicity, assume the number of cells dying is proportional to the total number of cells. So we have:
N = N0*2^(r*t) – rd*N0*2^(r*t)
Including the new rd term *does not* change the meaning of the original r parameter. The best fit value may change, but not the meaning. Even though the first model is way over-simplified, and the second is still wrong, the meaning of the original parameter does not change (r = divisions per unit time).
This is qualitatively different than the coefficient for “house size” in one of these arbitrary models.
Here is a discussion of the same distinction from the 1930s:
https://psycnet.apa.org/record/1935-02633-001
Thurstone and Gulliksen (one of his PhD students) are both great on this topic. As is, of course, Paul Meehl. None of it seemed to have had much effect on practice though.
Let’s try to be more practical and less preachy. I believe house prices are affected by size, location, features, age, type of construction. Is an estimated linear regression model (for example) a rational or an empirical equation? It would be nice to have a “theory” that leads to this estimation, but what sort of theory would make this rational rather than empirical?
I’ll admit from that lengthy excerpt you provide that I really don’t know what sort of theory would qualify in the social sciences. I could write a bunch of theoretical equations about how people derive utility from houses with a resulting equation to estimate, but that would only be a more mathy version of saying that these are the things that I believe influence house prices. Is there a rational theory that would qualify in your view?
Once again, your use-case for these coefficients is actually for the predictive skill of the model. Afaict, there is no actual use-case for the value of the lone coefficient. The counter-examples keep ending up not being the case.
What I am saying is very practical. Stop trying to interpret arbitrary coefficients, it is at best a gigantic waste of time. In fact, that might be the most practical advice possible to give on the topic of statistics over the last 70 years. Even more than to forget about statistical significance.
The solution is to instead optimize for out-of-sample predictive skill (weighed against the cost of developing the model). It sounds like this is already what you have been doing in practice anyway.
And this is where the world is headed (ie, machine learning) tin general. It took so long to overcome the inertia because the benefits needed to become undeniable first, which required cheaper computations. But we could have saved a lot of trouble by skipping the whole arbitrary parameter estimation thing from the beginning.
Just to be clear, are you saying that research such as this: https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.37.3.189 is a complete waste of time? This is a fairly standard example of economic research, though done better than most. You might particularly like the discussion of some of he empirical issues associated with potential biases in the regression coefficients. It sounds to me like you are saying that nobody should have been looking at those coefficients to begin with. Is that correct?
I just want to understand the scope of your comments. I’ll point out that the majority of economic research (probably the vast majority) has been a total waste of time if you are saying that any focus on model coefficients is misguided. I have plenty of criticisms of much of the economics literature, but even I wouldn’t go that far (and my concerns are mostly based on other things than the use of model coefficients). There have been some comments on this blog that seemed to suggest that much of the psychology literature (and probably almost all of the social psychology literature) is a waste of time, and while I share many of the criticisms I am still reluctant to write of another field of research so easily.
I do agree that data science is moving in the direction of predictive accuracy out-of-sample and is likely to continue in that direction. In many of these models, it would be almost impossible to focus on the model coefficients. But isn’t that a limitation of these models rather than a feature? The world is interested in understanding the influence of various factors, not just prediction. Of course, establishing causation from observational data is fraught with problems and perhaps impossible in theory. But insisting on controlled experiments for many questions in social science would end much (most?) of that research. The skeptic in me might easily say, “it’s about time,” but a more thoughtful response would be that it would be unfortunate to abandon any research where experiments can be done – particularly experiments geared to providing meaningful answers to such questions.
Just how much does the “world according to Anoneuoid” differ from the world I see around me?
Yes, this is exactly the type of thing I am talking about:
They are trying to estimate this gamma parameter and keep adding or removing other variables, then getting “puzzling” results all over the place:
That is because it is just an arbitrary value of a model with arbitrary structure (interactions, linearity, etc). If you wanted to use the same model to predict wages, it probably works reasonably well. To get at gamma, it needs to be on the LHS and measured somehow.
For 1000 years society assigned their most educated members to theological questions most consider meaningless today (angels dancing on a pin). Same thing is going on here. That doesn’t mean there aren’t interesting ideas to be found, but these coefficients don’t mean anything.
Anon:
Yes, we discussed this a long time ago on the blog: https://statmodeling.stat.columbia.edu/2004/12/30/what_is_the_val/
Anon
I guess we agree about the details but reach different conclusions. Specifically, you say
“That is because it is just an arbitrary value of a model with arbitrary structure (interactions, linearity, etc).”
In this particular example of compensating wage differentials, the literature you cite I take as evidence that the particular coefficients people are using to estimate (the value of “statistical” life) are unstable and unreliable. But I don’t conclude it is because they are trying to interpret the coefficients in their model. In some cases (the house value example, for instance), the coefficients are more stable and adding or removing variables, using nonlinear effects, etc. do not change the parameters substantially. When that happens, I think they are useful. I have examined many such variables – and I have examples where changing the model structure makes me wary of interpreting any of the specific coefficients. So, I think it depends on the circumstances and the data. Your pronouncement that all such efforts are doomed just doesn’t jive with my experience. Even when the coefficients change dramatically, there is often useful information derived from how they change in response to changes in the model structure.
That is perfect Andrew, from the abstract of the Dorman paper:
I was looking for negative gamma values in the Lavettit 2023 paper but didn’t see them mentioned. I know a priori that a whole slew of them exist, because that is how these arbitrary parameter values work. Selection bias at work.
They could go on for 1000 years this way, and never find the value for gamma. It is like thinking if you just keep measuring the height of enough people, the standard deviation (not SEM) will shrink to zero and you will finally know the “real” height of a human being. It is a quixotic quest.
Point me to a dataset on this. I will include/exclude whatever variables and use symbolic regression to come up with many, many models that fit the data just as well or better than yours, but result in a negative coefficient for one you think is stably positive or vice versa.
How will you distinguish between those and your chosen structural form and variables?
Here is another example of this I came up with last year:
https://statmodeling.stat.columbia.edu/2022/07/19/vegetarians-and-covid-after-adjusting-for-important-confounders-researchers-who-are-willing-to-make-strong-unsupported-conclusions-are-had-73-or-0-27-95-ci-0-10-to-0-81-and-59-or-0-41-95-ci/#comment-2066969
OK, try this data: https://data.transportation.gov/Aviation/Consumer-Airfare-Report-Table-6-Contiguous-State-C/yj5y-b2ir
The task is to model the determinants of average fares per route. I’ve used combinations of distance, passengers per day, market share data (largest and lowest fare), and various measures of the airline (usually major airline or other), and various measures of time. Sometimes I include interactions (with the time variable – for example, distance effects will vary somewhat with time as fuel prices vary) and sometimes not.
I think the distance coefficient is relatively reliable – you should try to get a negative coefficient for distance without building a ridiculous model. Market share is the most interesting variable because sometimes it is positive, large, and significant and it is fairly easy to get that to change sign. But it changes sign when you omit distance (because the most concentrated routes are generally the shorter routes). It also is positive and large in the earlier years and becomes negligible in the past 10 years. I attribute this to the large mergers that have taken place. I’ve even tried to model the effect of these mergers and it seems to support that conclusion.
From what you have said, all of these ideas I have are pure folly. I don’t think they are, so go ahead and show me.
So all I gotta do is feature engineer the new variable “flight duration” by doing t = dist/v, plugging in avg airspeed or something for v. I’m sure actual data that accounts for weather, boarding times, etc is out there but that should be good enough.
I can also get fuel price and consumption per mile data, and use that to engineer a fuel cost per flight feature.
If you think about it, neither the airline nor customer actually cares about the distance. That is just a proxy for other factors. It should be a very minor variable. I’d also guess fewer longer flights are cheaper for the airline (logistics, stress on the airframe) than many shorter flights. A large positive impact on price sounds like model misspecification.
I’ll play with the symbolic regression anyway and get back to you.
Anon
Now you aren’t even reading carefully. I never said the distance coefficient was large – it is a very small number but highly significant and fairly consistent in magnitude (more than you might expect given the variability of fuel prices, but most airlines hedge against that volatility). Also, inventing variables such as duration that might render distance to be insignificant is just a translation of the story – distance and duration are two ways to measure essentially the same thing – and if they are closely related enough then we are likely to get multicollinearity where I think we can both agree that interpreting a single coefficient is not appropriate.
And, if you want another example where regression coefficients are relevant, I’ll offer the Harvard admissions discrimination case. This section of the court decision – https://www.harvard.edu/admissionscase/wp-content/uploads/sites/6/2021/06/2019-10-30_dkt_672_findings_of_fact_and_conclusions_of_law.pdf (see section V)- I found particularly well reasoned for a legal decision involving statistical issues. Do you think the controversy surrounding the model specification and interpretation of the coefficients on race was foolish?
Dale, Anoneuoid has consistently said when the model has reasoned mechanistic components the coefficients can make sense. What he’s railing against is the “throw available information into a blender and see what pops out” style of linear regression fitting. In that context, it’s easily plausible for things like distance, and fuel consumption to both be in the same blender.
If we want to model what actually causes air fares, we’d want to look at the actual operations costs an airline has, as well as the value proposition to the customer. So for example costs could involve the fuel consumption, the personnel and the hours they spend providing the services, the airport fees for access to gates and landing slots and soforth, the wear and tear on the engines and airframes and tires and brakes and such caused by takeoff and landing, the effect of bad weather on aircraft performance and maintenance, as well as on personnel costs and delay related issues…
When it comes to the value prospect there’s the value of shortened travel times, the negative utility of having to deal with airports, security, drop off and pick up etc, the need to rent cars at the far end of the travel, the negative utility of wear and tear in personal cars from driving that’s saved by flights, the possibility to work on the airplane vs driving in the car, the negative health effects of sitting for a long time in the car, anxiety over flying, etc.
If you build a model on those physical and psychological determinants of value, your model is meaningful. If on the other hand you pick 6 variables and throw them in a blender, then you’ll capture some aspects of some of these determinants, and if you throw a different 6 variables in a blender you’ll capture different aspects… Each small number of variables projects the higher dimensional manifold onto some projective description and that projective description looks different in different sub-spaces, and hence the coefficients will move around a lot potentially.
When you’re in that scenario I think Anoneuoid is correct. On the other hand when you’re working with the more fundamental mechanistic model and particularly when you’re doing so in a dimensionless formulation… you have a much better chance of capturing a consistent viewpoint on the problem and having a robust model whose coefficients mean something.
Daniel
Economists generally refer to what you are stating as the theoretical model underlying the empirical work. I’ve had articles rejected for lack of a theoretical model – but it is mostly an empty criticism. I can write down a few equations regarding how airline costs depend on fueld costs, time (mostly staffing), and airport fees and taxes and a utility maximization equation where a representative consumer’s utility depends on time of travel, price, and some location characteristics. In the end, that “theoretical model” will generate an estimating equation with the same variables I have picked to put in my model based on my loose intuitive understanding. If all you and Anon are saying is that equations are better than this loose intuitive understanding, then I’ll agree but frankly, so what? If you are looking for something akin to a biological or physical mechanism that determines airline prices, then keep looking.
I am not saying just collect data (on shark attacks, sports scores, motel prices, home prices, ….) and throw it in a blender to see what comes out, then I think we can agree that such a model might be useful for prediction but not for interpretation, and any focus on the coefficients in such a model is futile. But I think our disagreement is about what constitutes an appropriate theoretical background or mechanism that should underlie any model. Apparently, neither of you believe that it is adequate to choose the features on the basis of some intuitive understanding.
This makes me wonder about Bayesian estimation – how do you decide what information is relevant for the priors? Suppose there are 10 related studies and one or two have wildly different estimates than the others for the effect of a variable. Do you include those exceptional studies in forming the prior or exclude them? Is there a model whose mechanism will tell you whether or not to include those disparate results? I’ve often seen Andrew specify that certain effects should not be larger than some value x – based on his intuitive understanding. I have no problem with that. If you don’t either, then how is that different than the models I build to explain airfares across different routes?
In hopes of resolving the discussion between Anon, Daniel, and myself:
I can see that modeling the mechanism whereby the costs of operating an airline on a specific route is superior to just saying these costs depend on x,y,z,… We can, in principle, specify what the cost function would look like, and this would lead to more specific form to estimate. In reality, unless we know something about how airlines have (or have not) hedged their fuel costs, we will have to decide what fuel costs (current, prior year, some mixture) we should use in the model. If our knowledge is vague enough, we might just be better using fuel prices as a variable in the (regression, for example) model and not attempt to specify how important the variable is relative to the others. In other words, a purely empirical approach might or might not be better than attempting to model what the cost function “should” look like.
When it comes to consumer valuation of air travel on a route, I think we have no choice except to use an empirical approach. If you are willing to specify what form the utility function takes, you could derive specific forms for the valuation – but I would not be willing to do that. So, I am willing to let the data lead to quantifying the relative importance of the things that would enter that utility function (distance, connections, population, location features, etc.). I really have no idea how to model the importance of these features without seeing what I can from the available data.
Now, if either Anon or Daniel are saying that attempts to interpret model coefficients are useless without being able to specify the utility and/or cost functions, then I probably don’t agree. But if we can agree that the more we can specify the mechanism the better, but in the absence of such knowledge a purely empirical approach can be used, then perhaps we agree. By a “purely empirical approach” I don’t mean randomly throwing variables into an estimation, but using what is available based on our intuitive understanding of what is important to airlines and their customers. And, if we use that empirical approach, we should be wary of how stable any of the coefficients are when adding or removing variables or changing the structure of the model. The more sensitive these coefficients are to changes, the less reliable any interpretation it might suggest.
Is it correct that you include largest and lowest fare in your model? So, besides those, the numeric features are year, quarter, nsmiles (the distance), and passengers.
I’m playing this package (never tried it before), which doesn’t do categorical by default: https://github.com/MilesCranmer/SymbolicRegression.jl
I still really have no idea what you think the coefficient for distance means in your regression. What are you using this value to do?
Dale, Anon, I’m going to reply at the bottom because the thread is too deep to easily continue here.
Anon
I don’t use largest fare or lowest fare to estimate average fare – the multicollinearity problem is obvious. I have used both of those market share variables and they appear to suggest some interesting stories about how competition works in the airline industry. I believe the coefficient on distance is a reasonable measure of the fuel costs – I haven’t checked that, but it should be able to derive an approximate value from knowing the fuel efficiency of aircraft. As Daniel suggests, it also reflects consumer utility and, to an extent, additional staffing costs. Those can probably be estimated as well (from a mechanistic approach) to see how much of the distance effect is not determined by operating costs. When I’ve modeled this data, my primary interest has been on the market share of the largest carrier – that coefficient is what I’ve focused on and so I am not too concerned about the exact interpretation of the distance coefficient (though it is quite stable across almost all specifications of the models).
Thank you for giving the lecture at Osaka university by the invitation of Professor Joe Suzuki. I would like to introduce that Bayesian statistics has two effective dimensions. One determines the marginal likelihood and the other gives the prediction accuracy. The former (RLCT) represents the speed at which the posterior distribution shrinks, and the other (Functional Variance) does the fluctuation of the log likelihood in the posterior distribution. Both are birational invariants.
Nowadays, machine learning researchers are interested in the fact that both are far smaller than the number of parameters in deep learning with Langevin or Hamiltonian Monte Carlo, which shows Bayesian statistics plays an important role in overparametrized systems. Your opinion will be appreciated.
I’m looking forward to hearing your lecture in the next year.
It’s so nice to see you on the blog, Prof. Watanabe! I would very much like to understand what you mean here, but I looked up “binational invariant” on Wikipedia and it’s beyond my math knowledge, as it says, “A birational invariant is a quantity or object that is well-defined on a birational equivalence class of algebraic varieties. In other words, it depends only on the function field of the variety.”
RCLT is “real log canonical threshold,” a measure of effective dimensionality developed by Prof. Watanabe for singular (i.e., non-identifiable) models: https://www.alignmentforum.org/posts/4eZtmwaqhAgdJQDEg/dslt-1-the-rlct-measures-the-effective-dimension-of-neural
We don’t allow non-trivial singular models in Stan because when a model’s not identifiable, the posterior isn’t a proper density and HMC will just run away and diverge. Trivial examples of non-identifiability are things like label swapping in a mixture model—we can deal with that in Stan as it still has a proper posterior.
I didn’t understand the point about Langevin or HMC and how it relates to ML.
Dear Bob, thank you for your question. Effective dimensions of Bayesian statistics do not depend on the representation of parameters. It is invariant under the birational transform of the parameter.
In machine learning, singular models are ubiquitous, for example, deep learning, normal mixtures, topic models, reduced rank regressions, Boltzmann machines, and so on. They are nonidentifiable and have singularities in the parameter space. In singular models, the maximum likelihood estimator (MLE) quite often diverges whereas the posterior distribution does not. Since the posterior distribution concentrates on the neighborhood of singularities, Langevin and Hamiltonian (No-U-Turn) MCMCs converge to the posterior distribution if they are properly adjusted. To check whether the posterior distribution was sufficiently approximated or not, Gelman-Rubin statistic and diagnosis developed by the Stan team are useful. Further advances of Stan and its diagnosis will much more benefit both statistics and machine learning. Thank you.
Do frameworks like VC dimension or Rademacher complexity (which tend to have more visibility within machine learning than within statistics, AFAIK) have a role to play here?
VC theory tells us that you can have a one-input binary classifier with one free parameter which can fit any finite dataset: sign(sin(alpha*x) ) where alpha is the free parameter. You can get the sinusoid to wiggle fast enough to fit any binary labelling of a finite set of x_i’s. Maybe that comes across as a contrived and pathological example, but it does demonstrate that “number of parameters” has its limits as a concept.
Conversely, the set of monotone functions has an effectively infinite number of degrees of freedom and yet is highly constrained. Reasonably tight guarantees about the difference between in-sample and out-of-sample error can be provided as long as the correlations between the inputs are not pathological. So that’s the flip side of the sign(sin(alpha*x)) example.
Dale, Anoneuoid. Re these posts: https://statmodeling.stat.columbia.edu/2023/11/30/effective-number-of-parameters-in-a-statistical-model/#comment-2293488
I’m somewhere in the middle on this question. I believe you should be able to specify a-priori what the important variables are. If you use those variables in a purely “curve fitting” type regression I think this is fine, and with a large amount of data is probably very good.
What I disagree with is throwing things into regressions on the basis of what’s “available” or some half-baked argument about “proxying for” something else where the connection doesn’t have a good theoretical basis, and also any argument about “unbiased estimates” or such which might lead you to accept a functional form in which the variable of interest could have an overall effect which is counter-to theoretical conclusions.
Let’s give an example: increased distance should make flying have more utility for the consumer and higher physical consumption of resources (fuel, personnel time) for the airline. So we should expect that increased distance over sufficiently large increases, to increase airfares. That is, whatever the nonlinear shape of the function, if d is a “short” trip then f(d+dd) – f(d) is positive for all sufficiently large dd. This automatically is obtained in a *linear* regression for a positive coefficient on d, but I have no special love for linear regressions, however, I still think in any such model a Bayesian prior should put high probability on “overall increasing” functions of d.
I also think that Buckingham’s Pi Theorem always applies. So if you have for example:
distance D (length), fuel consumption rate F (volume/time), flight speed V (distance/time), fuel cost per volume Cf (dollars/volume), and average wage rates W (dollars/time) then because there are 5 variables and 3 fundamental dimensions here (distance, time, and utility/dollars) there are two dimensionless parameters that completely describe the problem.
We’re free to manipulate them as we see fit but for example one such parameter is: F*(D/V)^2*Cf/W
Another such parameter might be: D^2 (Cf/W) V
So, throwing all 5 parameters into a linear regression is *guaranteed* to produce something with redundancy in it and confusion about what happens when you drop out individual variables.
Even if “Omniscient Jones” tells us exactly what variables to use, that still doesn’t tell us the functional form.
We can know initial number of cells and rate of division are important then try models like N(t) = N0/(r*t), etc. How can this get us to N(t) = N0*2^(r*t)? Or for volume of a box it could be v = h*l^2/w.
When we know the answer, I have never seen that fitting these linear regressions has been able to figure it out. It only works for toy examples where the data is actually generated from the same type of equation.
Once again, none of this matters if all you want is predictive skill. That is the use for these models.
Anon…
V = h*l^2/w is not dimensionally consistent.
It’s true though that we can’t get the functional form in the general case from the variables alone. However we can utilize general purpose approximators and with sufficient data get a function that is successful using an appropriate set of dimensionless groups.
However looking at single coefficients in universal approximators is a foolish thing to do. You are much better off describing the function behavior directly.
Also the function will be of a dimensionless group, not of an individual variable.
Without knowing any better, you could come up with a theory about “dark dimensions”, where height has special units of m^2. Or just define volume as having units of m^2. We do know better, but only because we already know the correct relationship.
But let me rephrase it then. Can you show how to start with a model like:
v = B1*l + B2*w + B3*h + [terms containing correlated but irrelevant variables]
And then conclude the actual equation is v = l*w*h?
Even better, a more realistic example would be the one on moons per planet I mentioned above: https://statmodeling.stat.columbia.edu/2022/07/19/vegetarians-and-covid-after-adjusting-for-important-confounders-researchers-who-are-willing-to-make-strong-unsupported-conclusions-are-had-73-or-0-27-95-ci-0-10-to-0-81-and-59-or-0-41-95-ci/#comment-2066969
Anon. I think there’s some confusion about what “dimensions” mean. Basically, two physical things have the same dimensions if they measure the same “type” of thing, and they therefore make physical sense to add together. If they are in principle measurable with the same instrument then they are of the same type.
If it makes no physical sense to add them, and they can’t be measured with the same kind of instrument, then they have different dimensions.
So when it comes to length, width, height, these are clearly of the same type, for example each of them is measured with a ruler.
When it comes to “volume” you could argue that we don’t know the relationship between dimensions of volume and dimensions of length, and we need to derive that relationship through some process. I think that’s reasonable to say.
I think you’re asking suppose we had measurements of blocks of material with length, height, width known and volume measured by say displacement of water in a graduated cylinder. How can you find the proper relationship?
One method would be to say that we are going to universally approximate the relationship… We’ll say that it’s a polynomial in each variable whose coefficients are each polynomials in the other variables.
V = A*P0(height) + B * P1(height) + C * P2(height) + D * P3(height)…
where P0, P1, P2, P3 are for example the basis polynomials 1, x, x^2, x^3… and A,B,C,D are recursively represented in the same way with similar polynomials in width, with coefficients that are polynomials in length…
You’ll wind up with a large number of coefficients etc and if you do a regression on this complicated form, you’ll find that all the coefficients are close to zero except the one for height*length*width
That doesn’t sound at all like the usual regression workflow.
It does sound like what they did to demonstrate that SymbolicRegression.jl package could be useful:
https://arxiv.org/abs/2202.02306
I’m somewhat skeptical of symbolic regression since there are just so many hyperparameters involved, but they show it *can* work.
I’d like to see it shown that the widespread practice of fitting arbitrary linear models is actually capable of finding the right answer. It seems to me the standard procedure was adopted with no evidence that it can work*, even in principle.
* Of course, if you simulate data from the same model, then it works. But that requires knowing the correct functional form to begin with.
The standard practice of fitting arbitrary linear models absolutely does not work in general.
The usual justification for linear models, if there is any effort made to justify it at all… Is the idea of Taylors series. In other words you’re discovering an approximation to the real function in the vicinity of certain values for the inputs.
Most people taking a stats class never got that part of the theory.
It sometimes feels like we are descending into a dark ages where PhDs are sent out into the world knowledgeable about religious rituals and no one understands the underlying logical basis (or lack thereof). As soon as you’ve learned a complicated ritual in place of a logical framework… It’s ripe for people to be doing illogical stuff and teaching that to their own students for a couple generations, and soon… All real knowledge is lost.
Trying my best to counteract that in my own book by completely reorganizing the thought process.
Daniel
I don’t disagree with you. But I’d point out that advice is often given (I’ve seen it on this blog) to use a linear model with a nominal response variable rather than a logistic model. Despite it being an incorrect (wrong) model, it is sometimes useful. My only point is that taking an extreme position against linear models conflicts with the practice followed by many professionals, including some that are knowledgeable. I’m not advocating for linear models, but I am leery of taking extreme positions.
The linear models are useful, for making predictions.
Trying to extract some meaning from the coefficients is a quixotic quest though. This isn’t extreme, it is reality. Just because lots of people are confused and/or do the wrong thing, doesn’t make it extreme to point it out.
No matter how many resources society pumps into praying monks, they will not invent and build a smartphone. Devote everything to it, for millenia, still no. The method simply does not work for that purpose, widespread belief otherwise is irrelevant.
A popular term that symbolic regression comes up with is -5800 modulo year. Why?
For 1996-2023 that corresponds to avg fare increasing ~$3 per year, from ~ $190 to $270.
Thats interesting, but -5800 is a completely arbitrary value that happens to fit the data for that given model specification. It would be very silly to calculate CIs and check if its different for different airlines, and so on.
Anon
I’m afraid I don’t know what you mean by “symbolic regression” or “modulo year.” But your comment just seems silly here. The coefficient on year in a linear regression model for average fares (with the other variables – distance, passengers per day, market shares, and perhaps major airlines) is meaningful. I don’t know what your point is. Year is meaningful and the time trend tells us something (although if you look at the data, a linear trend certainly doesn’t capture the movements within the whole time period available). Whatever the -5800 means, is your point that you can put variables into a regression that are meaningless and then interpreting their coefficients is meaningless? If so, I agree with you completely. But that is hardly worth a comment on this blog. So, I assume I just don’t understand what you are saying.
Dale, in Symbolic Regression instead of trying to find the “best” values for numbers (coefficients) in a linear expression or even a nonlinear expression, you instead attempt to find the best expression overall. The system operates on parsed computer expressions and works by modifying the parse tree to search for a functional form that predicts the data.
I can only assume that the process Anon is running is somehow coming up with the subexpression (-5800 % year) where year is a variable holding the observed values
Modulo (mod(x, y) in julia) just means the remainder after dividing x/y. So 10 mod 5 = 0, 10 mod 6 = 4, etc. Here are the most recent results I got. IT shows the best fit equation (using mean squared error) for given equation complexity (they really, really depend on hyperparameters):
https://pastebin.com/PEJsVRhy
This was the code I used:
https://pastebin.com/rJZwwDik
Anon
Sometimes I think we are on different planets. I know what the modulo function is. What I don’t know is the technique you are using to fit a model – and I don’t really want to know about it. It looks to me like just another automatic data mining method, perhaps more sophisticated than stepwise regression or any number of alternative mechanisms. I do rely on automatic (or at least semi-automatic) methods for feature selection sometimes, mostly when there are too many features available. But I’ll be the first to admit that I have no idea how this post relates to the question of whether it is foolish to try to interpret model coefficients.
Also, the usual regression fitting algorithm only fits the coefficients/parameters. Symbolic regression also fits the *operators*. Ie, it tries changing out “+” for “*”, “/” , “-“, “^”, etc to see if there is a better fit.
Dale, again I seem to be living on the planet in between the two of you. I’m not quite sure what Anon’s point is exactly except that a part of the point appears to be that he’s seeking to invalidate the idea that coefficients are interpretable by examining the airfare problem and having a Symbolic Regression process find some nonlinear combination of some of the variables which fits as well or better but switches the sign of the distance coefficient or otherwise contradicts the conclusions from a simple regression, to show that unless you *know* the form of the true generating model you will usually be “reading tea leaves” when you throw even *known important* variables into a simple linear regression.
I would say he’s likely to get farther if there are significant nonlinearities in the relationships between the variables. There are probably a lot of cases where the Taylor series justification holds sufficiently well that 1st to 3rd order polynomials are sufficiently good to fit the problem over the realistic range of the covariates.
I don’t think anyone disagrees with Anon saying that if you have a mechanistic reason to believe the formulas you derived are correct then the coefficients are interpretable. I think you believe that there are plenty of cases where the coefficients are robust to specification differences and are therefore interpretable outside the mechanistic setting. I think everyone including you agrees that there are plenty of “non-robust” problems where throwing stuff in a blender and trying to drink the milkshake that comes out doesn’t work.
I’m not sure if Anon believes the strong form of his thesis which is something like “unless you have a mechanistically derived model, the coefficients are *always in every case* meaningless”. If so, he may be trying to demonstrate that Symbolic Regression can always transform the problem in such a way that it fits *better* but effectively has “noticeably different coefficients” compared to someone elses given representation.
This is plausible. But, can you show a single example where this method got what we know is the “right” answer? Given the popularity of these methods, this must exist by now right?
I want to see an example of it actually working.
It is just taking the standard methods you are familiar with to the logical conclusion. Why stop at fitting the coefficients when you were going to use an arbitrary model to begin with? Why shouldn’t we also fit the relationship between the variables/coefficients?*
It is relevant because if you play around with it then the arbitrariness of your choice of model will become undeniable. It is just that you “like” y = B1*x + B2*y + …, etc. And you only like it because that is how you were trained, and you were only trained that way because it was hard to work out more complicated models when the textbooks were written.
* The other (much preferred) alternative is to derive a model from some plausible assumptions.
> It is just that you “like” y = B1*x + B2*y + …, etc. And you only like it because that is how you were trained, and you were only trained that way because it was hard to work out more complicated models when the textbooks were written.
This is more or less true for a lot of people I agree. But I think there’s more to it. Even if you want a nonlinear function of say y, you can represent this as a basis expansion in some basis of nonlinear functions, like for example Chebyshev polynomials. Let’s call those T0, T1, T2, T3 etc. then:
A0*T0(y) + A1*T1(y) + A2*T2(y) + … + AnTn(y)
is a truncated chebyshev series in the variable y and is *linear* in the unknowns A0…An, furthermore if the function is a continuous smooth function of y on [-1,1] then the series converges exponentially, so the maximum error between this function and the “true” function goes to zero like exp(-k*n) for some k. That’s really fast, so that often only 5 or 10 terms are needed for even highly nonlinear functions. To see this you can try creating your own weird nonlinear function of 1 variable and then asking ApproxFun.jl to approximate it.
Treating a function space as a vector space is a well established applied mathematics “trick” which converts nonlinear problems in y into linear problems in coefficients of basis functions. So it’s not entirely arbitrary that people focus on linear fitting.
On the other hand, when there are *multiple* variables of interest and the problem is nonlinear in all of them, like sqrt(a*b)*c*atan(c*d)*exp(-r*d) or something then you have a very hard time representing that in terms of series on a,b,c,d it’s not at all efficient.
This may or may not actually work on real data, I don’t know. But, if you only have arbitrary data laying around then even if that method works the model will still be arbitrary. Eg, you have distance between cities, but not the more relevant flight duration or fuel consumption.
Dale:
Another thing I was thinking about, the reason I drop highly correlated features is due to computational efficiency. What is the justification for throwing out info when that is not a concern? Eg, using a linear model that has <10 terms?
Wikipedia says (I removed the equations):
https://en.wikipedia.org/wiki/Multicollinearity
I don’t see this being an issue for real-life measurements.
Anon
We are reaching diminishing returns in this exchange. But I’ll respond to a couple of novel ideas you have raised here. Regarding multicollinearity: it is well known that the predictive power is not affected, but the coefficients are likely to be. I have cases where strong enough multicollinearity (not perfect however) in fact renders the coefficients meaningless (both size and direction are wrong).
I don’t disagree with the superiority of deriving a model from principles rather than searching whatever data you have for something that “fits.” However, in most of the problems I analyze, there is relatively little structure available to derive a model from theoretical principles (unlike many natural sciences). In the airfare case, I do think a model of airline operating costs can be so derived. In the case of consumer valuation, I don’t think so. I could specify a utility function but that is arbitrary and likely worse than using some empirical fit to data. Should that empirical model have nonlinear terms, interaction effects, or additional features? I’d let the data guide me in that. I’ve seen Andrew suggest including all interaction effects in models – but it seems he means all “reasonable” interaction effects (I still don’t think you can include all interaction effects).
Every time you add there additional terms, the model coefficients are likely to change – but the degree to which they change varies. Sometimes, the changes in response to these additional terms is itself interesting, and when the coefficients do not change, I gain some confidence in them (regarding sign and size). You appear to be saying since you can generate a mathematical form that could change the sign or size (substantially) that it renders the interpretation of the coefficients foolish. I don’t agree – it depends on how reasonable those mathematical forms are (e.g., using 6th degree polynomials fit to data I generally think is not reasonable – there are few cases where a reasonable process would generate a 6th degree polynomial as the “correct” model). I’m willing to use judgement regarding what is “reasonable” if I have sufficient subject matter knowledge (which I only have for relatively few subjects).
Model A tells you the effect of distance while “adjusting for” year, etc.
Model B tells you the effect of distance while “adjusting for” year, fuel consumption, etc.
The coefficient means something totally different, that is why the value changes. What basis is there to use Model A rather than Model B? It gives the coefficient for distance if we ignore a bunch of stuff that actually determines the price of the tickets.
Anon
The justification for using the distance coefficient from model A would be that it has a value very close to what you get from model B (though model B is still a better model, just not telling you much more about how distance affects fares). At the same time, no version of model B will be immune to the criticism that there are omitted factors. For most social science problems, I suggest that there is no complete list of factors that can be specified – does that mean we can/should never interpret the coefficients of any model?
While prediction is one important purpose of such models, the coefficients are important for many policy (public and private) issues. How much do years of education vs years of experience contribute to wages? Do men have different earnings than women, after adjusting for appropriate factors? What impact does market share have on airfares (important for analyzing potential mergers, such as the impending Alaska Airlines/Hawaiian Airlines merger)?
All of these questions could be analyzed by building a model specifying the process by which wages, earnings, airfares, etc. are determined by factors x, y, z, etc. In many (most from my experience) cases, those processes are not well understood and a theoretical model that specifies the form of the estimating equation(s) would be hard to justify. I might prefer to choose factors and specify models based on empirical evidence. We can and should evaluate the quality of that evidence (like staying away from 6th degree polynomial fits).
Dale.
In the presence of ONLY the distance variable, the distance coefficient will play the role of a method of accounting for costs that grow with distance or flight time… fuel, personnel wages, wear and tear, etc.
In the presence of distance and fuel consumption, the distance coefficient will not do that because fuel consumption is closely related to distance, and so the way these two split the difference is determined by essentially model error and measurement error and random noise in the data.
to first approximation, fuel = consumption_rate * distance/velocity, and consumption_rate/velocity is a constant that depends on the airplane and route, so fuel and distance are closely linearly related quantities, they’re kind of like:
cost = … + a * distance + b * fuel == … a * distance + b*velocity/consumption_rate * distance
so in the absence of fuel, your coefficient will change from a, to (a+b*velocity/consumption_rate) approximately.
So far, so pretty much standard. But add in nonlinear effects and then you’ll get different shifts depending on which range of covariates you look at because the “best fit line” when the true response is a nonlinear curve depends on which section of the curve you’re looking at.
All this is to say that looking at the *coefficients* is usually the wrong way to go about understanding a model, and also that you should do dimensional analysis on your models before plugging things into your equations, because what matters is *always* dimensionless ratios, not dimensional measures like distance, or fuel consumption.
Daniel
I hate to prolong this discussion but I still don’t get it. In the data, I think it is reasonable that when measured across routes, your term velocity/consumption rate may not vary much. So, I don’t think that term really changes the effect of distance on cost. Also, the presence of fuel prices introduces another complication, particularly if the data is time series (where those prices change a lot). I really don’t get your point about adding nonlinear effects. If there is any use for NHST it may be to help decide whether nonlinear effects are important in your model. After all, there are an infinite number of nonlinear effects you could put in a model – to decide not to use these either requires enough understanding of the process (which might be possible for the cost function, but certainly is not for the demand model) or some evidence from the data. While I know that a very high p value for the nonlinear terms does not mean that they don’t matter, I think it is legitimate to not include them when the data suggests that there is too much noise to identify the effects. I also don’t understand your point about dimensionless measures – there is nothing wrong with those, but provided that you keep track of the units of measurement, I don’t see why it is “always” dimensionless measures you must use.
I think the positions you and Anon (I see some differences but in this respect I’ll treat them the same) are espousing are at odds with virtually all social science research that I am familiar with. Issues like determining the extent to which wage discrimination exists don’t follow what you are saying here (unless I am completely missing your points). While the entrenched practice of p-value cutoffs is widely practiced and deservedly criticized, I don’t believe that all of the economic research is completely doomed. Here is a concrete example: https://link.springer.com/article/10.1007/s10663-020-09492-4. While I don’t think this is the place to critique this study (I’ve only browsed it), I am wondering if you think this is the kind of study that shouldn’t be done. After all, it is focused on various coefficients that affect wage gaps, and it has not included all possible nonlinear effects (beyond using logs).
Dale, first off let me complain that I substituted the wrong thing it should be consumption_rate /velocity everywhere since we are just substituting for fuel consumed… The perils of doing math in comments while distracted by kids.
The point about this not varying much seems strange. It’s literally a linear function of distance.
If you leave out fuel, you will get a coefficient for distance which is bigger than the coefficient when you include fuel as a separate regressor because the fuel is also a linear function of distance. If you include fuel explicitly then distance coefficient will mean “all the stuff that scales linearly with distance except fuel” whereas if you leave fuel out of the regressors then it will mean “all the stuff that scales with distance including fuel” the meaning of the coefficient changes a lot between those two models.
If there are nonlinear effects, like for example the plane needs to dump fuel to land safely, then depending on the distances you’re sampling you will get different values for the distance coefficient. For example if you have a bunch of small to medium distances you’ll get a steeper coefficient than if you include some coast to coast flights or Hawaii flights because they will never be dumping fuel on those flights.
The point about dimensionless coefficients isn’t about keeping track of the units it’s about eliminating redundant symmetries in the problem. An example, suppose you have calories burned, height of subject and distance walked…
If you regress C ~ H + D you have failed because there is a redundancy in this problem. The problem as stated is completely determined by only ONE dimensionless measure, namely D/H
This failure adds to the confusion caused by throwing things into regressions and is common, exceedingly so.
One major problem is that in more complicated examples the dimensionless formulation will show you that the determinants of outcomes will involve dimensionless groups with certain covariates raised to very specific powers other than 1
In an airline example just to construct something that illustrates the point perhaps one variable might be the unloaded mass of the airplane, and another variable is average height of the population at the departure country (international flights). Well to form a dimensionless ratio we should do something like M/(n*rho*h^3) where M is unloaded mass of the airplane n is number of seats, rho is the density of human bodies and h is the height of citizens in the country of origin. Essentially this will be measuring fuel consumption issues caused by heavier occupants.
Dimensional consistency requires us to use a specific power of the height.
I only skimmed the paper you mentioned I’d guess they have a dimensional consistency issue with regard to currency measures but knowing economists they probably have back-doored that issue with “inflation adjusted dollars” or some such. IMHO they could probably benefit from dimensionless measure of number of children, basically dividing the number of children in each age bin by the number of adults in the household. The multigenerational household is a real and important way that some families deal with child care. The burden of extra children is different in households with 2 grandparents living in.
I suspect the fraction of microeconomics who even understand the existence of the issue I’m discussing is tiny.
“If you leave out fuel, you will get a coefficient for distance which is bigger than the coefficient when you include fuel as a separate regressor because the fuel is also a linear function of distance. If you include fuel explicitly then distance coefficient will mean “all the stuff that scales linearly with distance except fuel” whereas if you leave fuel out of the regressors then it will mean “all the stuff that scales with distance including fuel” the meaning of the coefficient changes a lot between those two models.”
I really don’t see the problem. Interpreting the coefficient as “all the stuff that scales linearly with distance except fuel” in one model and as “all the stuff that scales with distance including fuel” in the other seems perfectly understandable to me. The fact that the numerical magnitude of the coefficient varies is not a problem – after all, it has units of measurement attached to it. So, is the problem you and anon seem to keep raising that the numerical magnitude varies with the model? Surely it is more fundamental than that?
Dale, it’s not just that the numerical magnitude varies with the model but that the *meaning* varies with the model, and yet, also most interpretations will give interpretational significance to the magnitude and even compare the magnitudes between models without giving serious thought to the meaning.
Furthermore, this is done in contexts where there’s far less clear meaning than the distance example. What is the meaning of the marginal utility of a dollar to a household in Michigan “controlling for” peanut consumption vs not controlling for peanut consumption? How about controlling for number of un-occupied dwelling units? Or the price of coffee, tea, and other beverages? How about something like “the effect of policing on violent crime” controlling for latitude vs not controlling for latitude? Or heating degree days, or distance from the Mexican border, or the price of childcare, or …
And so we continue…
It looks like your position and anon’s may differ. You are objecting to the poor practices of many researchers. Yes, the meaning and magnitude of the coefficients will differ depending on what is included in the model. I have no problem with that – the meaning and magnitude should matter, and these changes are in fact what may be of interest. It seems that anon goes further – there is no point to interpreting the coefficients at all (unless they are derived from a known correct model of the mechanism). You seem to agree with that for some problems, such as your marginal utility of income examples. I can also agree that there are some examples where it is meaningless to interpret such coefficients. But some does not equal all.
Further, the examples about the marginal utility of income (assuming there is a way to measure that) varying depending on whether or not prices of peanuts, coffee, tea, etc. are “controlled” for assumes that there is a large difference in that measure. There may or may not be. When additional factors are included in the model and the coefficients change a lot, I take that as important information. Usually there is a story worth telling in those changes, and it certainly precludes careless interpretations of the coefficient as the true impact of that factor. In other cases, the coefficient may not change much, and then I feel more inclined to interpret the coefficient.
What exactly is objectionable in what I have said?
As expected, it was trivial to turn the distance coefficient negative.
I simply looked up that a flight from London to NYC is ~$55k in fuel and ~3500 miles (~$15/mile). I then featured engineered a fuel cost variable by assuming that cost is increasing about $1 per mile per year since 1996 (the start of your data) and added some gaussian noise.
Adding this new variable switched the distance coefficient from a highly significant .054 to a highly significant -0.007:
https://pastebin.com/p92C7Dga
Above you said:
Pretend that was actual data, and not a crude guesstimation. Why would the new coefficient be meaningless/wrong but the original one was ok?
NB: The second model would be based on *more* information. It seems somehow your procedure punishes people for including more info. This seems bizarro to me like the rest of the NHST ecosystem.
Dale I think we agree there in principal, but I don’t know whether we would agree where to “draw the line” so to speak in individual cases.
In essence your objection comes down to something like “if there is an informal mechanism-like basis for the model, and if in practice the models coefficient of interest doesn’t change when other variables enter the model, then we can treat the coefficient as if it were approximately a constant describing a sort of law of nature”
I think far too often people proceed without an effort to justify why models take the form they do, and almost never in dimensionless form which eliminates some of the problems of unanchored coefficients. I guess I’ve been shown some macroecon dynamics models that proceed in dimensionless form but I don’t think I’ve ever seen a microecon example other than to use “constant dollars”, interest rates, and things like labor force participation or unemployment rates which naturally already have a dimensionless form. These are usually not conscious choices by the analyst, but rather just using standard measures in already dimensionless form.
The problem is equally common in areas like medicine we saw it in the vaccine trials for RNA covid vaccines. There were 3 age groups: 12+, 5-12, and 0-5 years, they used typically like 3 different doses (measured in micrograms) plus placebo in each group and looked at the difference in outcomes between the groups. Of course the determining factors of how the immune response proceeds must be dimensionless ratios. The alternative is that the way you measure the dose matters, that some choice of what a kilogram meant back in the days of Napoleon is a determining factor for how the immune system responds to a covid vax today… To understand this, consider we maybe inject 10M molecules. We can measure this in say micrograms, which gives us a number which is the ratio of the mass of 10M molecules to the mass of 1 millionth of a gram mass. Now back in Napoleon’s time someone decided a gram was the mass of 1 ml of water and then it’s been modified but the definition kept approximately constant. If they had decided that say a gram was the mass of 10 ml of water, we would today be injecting an amount which measures 1/10 as much in “alternative universe micrograms” even though it’s the same 10M molecules… If you think that you could inject 10M molecules but *had someone chosen a different standard back in the 1800’s* your body today would respond differently to the injection… I have a bridge to sell you in Japan.
No effort is made at all to account for this though. For example measuring the mass of the human body, or approximating at least the mass of the deltoid muscle in which it’s injected, or estimating the mass of the immune system housed in the bone marrow, or somehow measuring the rate at which the body degrades the RNA in terms of micrograms per hour as a function of age and then including some time constant into the measure, or otherwise attempting to create a scale-free measure of how much was injected.
That lack of scale-free measure resulted in many many 12 year olds having severe side effects from what was obviously too big a dose, as the amount injected divided by the mass of immune system cells that could respond to it was much higher for 12 year olds than for 30 year olds, who were all munged together into the “adult” trial.
We didn’t “give the same dose to multiple people” we gave *some highly variable dose* as measured by the determining factor which is a dimensionless ratio of the grams injected to some important gram measure of the body it was injected into.
So if even in pharmacology we aren’t in the right ballpark for understanding basic science, we are definitely far from doing it right in most microecon applications.
In essence much of science today has regressed to a pre-scientific state because statistics as a push-a-button take-a-pill check-boxes-on-a-checklist type system has been adopted instead of basic principles of science which were known 120 years ago… we’ve spent time teaching people things like running linear regressions without teaching them things like dimensionless ratios, basic ideas in dynamical systems, or algebraic model-building by which they could create mechanistically justified choices of model.
If you want to see this in action just look at the last century or so of the history of the use of Body Mass Index (BMI) a dimensional ratio measured in kg/m^2 and thrown into a lot of different regression blenders over the years. These coefficients are meaningless in precisely the way that they predict that *had the French Academy of Sciences done something different in the 1790’s your healthiness today would be different*. It can be proved mathematically that the only way to eliminate such insanity is to build your model from ratios where the dimensions cancel out, so that whatever the people did in the 1790’s it affects both the numerator and denominator in the same way.
Anon
I get the same results as you show. But your cost variable is inappropriate – it is a crude estimate of the cost for the plane, not accounting for the number of passengers. If you divide that total cost (which is sort of an average fuel cost for the route) by the number of passengers (to get it on the same sort of per customer basis as the average fare), then the distance coefficient is highly significant and around $0.032. The fact that it is quite a bit lower than the $0.053 from the model that only has distance is due to the fact that the more complex model distinguishes between an estimate of fuel costs and a distinct miles effect (possibly due to additional costs due to staffing on longer flights or perhaps higher airport fees for airports on longer routes which are generally in larger cities).
My point is that the progression of increasing model complexity and how the coefficients change is often instructive. And we can evaluate the models – in this case, the more complex models are better than the simpler one. Much better models are available, and I think you will find that distance continues to have a significant positive relationship with fares. Again, all you are showing me is that you can generate models that give ridiculous coefficients for distance, from ridiculous models. I don’t see how this proves your general point that it is foolish to interpret the coefficients in models.
Daniel
So, what is your take on the Harvard admissions case I linked above (https://www.harvard.edu/admissionscase/wp-content/uploads/sites/6/2021/06/2019-10-30_dkt_672_findings_of_fact_and_conclusions_of_law.pdf)? In particular, the discussion beginning on page 62 which discusses the competing models. This is fairly standard microeconomics and an unusually astute judicial opinion (in my opinion). What modeling approach would you recommend?
Looks like the opposite, longer flights are much cheaper for the airline:
https://simpleflying.com/short-long-haul-flights-economics-guide/
Elsewhere, I read the impact of fuel cost on fares is nonlinear. Ie, when the cost of fuel rises so do fares, but they don’t drop back down when the cost declines.
You could also account for those factors and get a whole new value for the distance coefficient.
As I mentioned above, the actual distance plays essentially no role in the costs to the airline or what a person is willing to pay. It just happens to be correlated with the other factors that do matter. It is like including “year”. The value for year itself has nothing to do with the price. Year is just correlated with inflation and so on.
I still don’t understand how you are applying this coefficient to a real-life problem. Your examples of “use” keep being about devising narratives regarding the model or the coefficient itself.
> But your cost variable is inappropriate – it is a crude estimate of the cost for the plane, not accounting for the number of passengers
Dimensional analysis strikes again. The fare is in dimensions of dollars / person, the distance is in dimensions of length, the fuel cost variable is in dimensions of dollars.
There are three variables, and three dimensions (dollars, people, and length). According to Buckingam’s Pi theorem, the appropriate model is determined by 0 dimensionless ratios. which is another way of saying we don’t yet have enough variables to determine the relationship. This is because without a second variable involving dimensions of length we can’t form a dimensionless ratio that eliminates the dimension of length. One useful way to handle that would be to introduce the airplane’s average velocity… but that’s in dimensions of length/time so we introduce another dimension! We’re still not done… But we can introduce time reasonably through the typical passenger’s wage, in dollars/time and we can account for the number of people on the plane through the occupant count N.
Our new variables and their dimensions:
Fare F (dollars/person)
Distance D (length)
Fuel Cost Fc (dollars)
Velocity V (length/time)
Wage W (dollars / time/person)
Occupant Count N (person)
We now have 6 variables and 4 dimensions, so we can form 2 dimensionless ratios to find a relationship between them:
(N*F/Fc) ~ Q(W * (D/V)/F))
This relationship says that the total income from the flight as a fraction of the fuel cost is some potentially nonlinear function of the wage paid times the time taken for the flight as a fraction of the fare.
Mechanistically this makes sense… on the left is the multiples of the fuel cost we need to bring in, on the right is the multiples of the fare that a passenger “spends” sitting on the flight.
“actual distance plays essentially no role in the costs to the airline or what a person is willing to pay”
Anon – your linked source does not state the first part – it only gives some considerations that affect short-haul vs long-haul costs. It says nothing about the second part.
The use case is in terms of evaluating airline mergers. The primary effects of mergers is to potentially increase efficiencies (higher occupancy rates in flights, potentially more efficient planes made possible through consolidation) and the potential to extract higher fares due to decreased competition. Almost any airline merger (for that, almost any merger) has these two primary impacts: the potential for operating efficiencies vs the potential for higher prices due to decreased competition. Evaluating a merger is a complex matter that would involve the specific factors that would be affected. The coefficients on these factors would be the way I would try to evaluate these impacts. How would you evaluate whether a proposed merger will increase or decrease fares?
Dale, regarding the Harvard decision. I did grab the PDF and I started to wade into it and my eyes glossed over. It’s a huge wall of text and could use a lot more graphs. I didn’t see a clear description of the model in skimming. I basically don’t have the energy to try to figure out what they actually did. They describe some kind of dummy variable logistic regression, this sounds like “standard” social sciences type models. They look at questions of whether some coefficients regarding race are positive or negative in terms of deciding something that I’d interpret as: “if Harvard knew the race of the student, did that predict an observed increase or decrease in the frequency of admissions, for each race”.
Basically in these scenarios I’d like to see a careful discussion of why a logistic regression is appropriate, what interaction terms are relevant, why these things are appropriate, what the expected magnitudes might be, prior specifications, and a Bayesian posterior over the coefficients. Instead what I saw was very basic description of “what is a model” kind of stuff, obviously aimed primarily at lawyers with zero math background. I admit to not wading in very deep before my eyes glazed over.
One thing about linear models on indicator variables is that there’s only 2 levels of the variable 0 or 1, so there’s no real nonlinearity issues, it’s not like people are classified as 38% Asian or something, either they’ve checked the Asian box or not. I think it’s possible I’d be OK with some of these models, it’s also possible I’d be ashamed to read the details and see what kind of garbage people are doing.
It’s hard for me to get worked up about the question of admissions to Harvard. I’d personally just reclassify Harvard as a Hedge Fund / Private Equity firm and then tax them at about 90% marginal rate on their income (and do the same for all the major Hedge/Private Equity firms). Of course I’d say do the same for USC and Stanford, and Yale, and all of them. They’re not educational institutions except incidentally for the purpose of avoiding taxation. The question of admissions is like about 85th on my list of questions about these institutions.
Yep… age old story. Two different arbitrary models have different arbitrary coefficients that *seem like* they mean the same thing, but actually don’t.
Then there is no way to tell which of these models is “better” other than see which gives you the “right” answer. I didn’t see where out-of-sample predictive skill was even discussed!
This process only measures a wealth/power weighted collective prior. The data is actually rendered irrelevant. It is like finding bible quotes to support whatever it is you already believe.
I’d like to see something like “separation of science and state” so these kind of arguments are treated however religious arguments get treated by the court.
Use the entire model (including as much info as possible) to predict the fares. Ideally, trained/fit to data on previous mergers.
You need not only a model of how the number of airlines would change, but also a model of what would happen to the other inputs to the model… if there are various inputs to your model and you are at point (a,b,c,…q) before the merger then after the merger you need to predict what point you will be at (a,b,c,…,q) + (da,db,dc,…,dq) so that you can then evaluate the output of the model at that point. If you merge two airlines then it’s possible things could change like airport fees, cost of maintenance, and of course market power for airline employee wages and airfares among other things.
Sure, but it sounded like we only have eg, the ~10 variables in that dataset, and don’t know anything else about how airfare and mergers work. Then I would still use the whole model to predict, rather than use one coefficient.
Eg, I would fit the model then use it to predict what would happen if there was a new larger carrier with whatever marketshare. It still isn’t clear to me how this arbitrary coefficient for distance was to be used though… It has nothing to do with the problem at hand afaict.
But as much domain-knowledge as possible should be built into the model (and not just what features are included). It sounds like to get what you want you’ll either be making lots of dubious assumptions or sending someone out looking for *way* more data on previous mergers though.
It’s not the coefficient on distance that applies in this use case – it is the coefficient on mergers. You said you would look at previous mergers – so would I. My first inclination would be to use a dummy variable for the major mergers indicating before and after the merger (potentially with some time lag). Also, as Daniel indicates, several things change, even within the given data set. Market share (on some routes for some airlines) changes by definition. Passengers per day would also change – the coefficient for that variable is a collection of a number of factors related to airline operations (size of plane, load factor, airport fees, etc.). I would also use the full model to predict fares – the coefficients are used, but indirectly in deriving these predictions. Perhaps you view that use of the coefficients as not interpreting them, but I think that is more semantics than a real difference – either we are using the coefficients on market share and passengers to capture merger effects, or it is the coefficient on the dummy before/after merger that we are using to gauge the effect of mergers on fares.
The proposed Alaska/Hawaiian merger is for $1.9 billion. It has been awhile, but it used to be that consulting fees on mergers were worth around 10% of the merger value. If so, I’m willing to work on building a really good model for $190 million. As it is, I’ve built lots of models using this data but only for classroom use. I’ve not written any papers or tried to publish anything on airline pricing. So, while many of the ideas you and Daniel have presented are good ones, I prefer not to let the perfect be the enemy of the good. I’ll stick to interpreting model coefficients with a healthy dose of caveats. When I look at Andrew’s Regression and Other Stories book it has many examples of interpreting regression coefficients – admittedly with plenty of cautions and caveats (which is appropriate). But wholesale rejection of interpreting these coefficients is not something you’ve convinced me of.
The prior thread got too long, so here goes:
Both you and Daniel don’t like the approaches in the Harvard case. Fair enough, but you are both living in a different world than I am. Policy issues arise, legal cases are cumbersome, analysis is often done with the data at hand which is never what you really want, and wishes for predictive accuracy are often just that – wishes. I fear that in the real world – such as the Harvard case – both of your suggestions would result in the decision being made on the basis of political connections and/or gut feelings. The analyses are far from perfect, but at least they are looking at some evidence, and I think they are at least addressing some of the real issues associated with admission decisions. You are both free to disagree (and I suspect you do), but I think further debate on that it fruitless.
A bit more on airline costs. I don’t vouch for these analyses, but I just wanted a ballpark sanity check. This paper has some attempt to model air travel costs: https://www.sciencedirect.com/science/article/pii/S136192092030715X and here is some rough data on possible fuel prices: https://www.flightdeckfriend.com/ask-a-pilot/how-much-does-jet-fuel-cost/. Using some of the results in the first paper along with the prices in the second, it appears that each mile adds around $0.10 to total fuel costs (not quite linear, but not too far off) for a 150 seat jet, and about $0.06/mile for a 250 seat jet. These numbers are in the same ballpark as the $0.05 coefficient from the linear models. We can quibble or even have major debates about the credibility of these models and my very very rough guestimates, but I am inclined to think that the coefficient from the linear models is not a bad first approximation.
Can we do better? Absolutely. But I’ll give the same response to your reactions to the Harvard case – I’d rather not let the perfect be the enemy of the good, or even the enemy of the less than good. I’d rather be loosely correct than precisely incorrect. Of course there are limits to that statement and that is where the interesting discussion lies.
This is what your method amounts to, the court “preferred” one model over the other for whatever reason unrelated to the data. And they only saw two models. Imagine if the judge was exposed to the true diversity of equally plausible models…
It is better to be honest rather than slapping a fake veneer of science on it (exactly like veneers of religion you get from selective bible quotes/interpretations).
For example, if I really cant make a rational decision about something, I have two magic 8-balls in a shoe box. I ask them rather than lie to myself about it. And yes, they have “saved me” from natural disasters before.
Dale, for $150M I’d be willing to build a very good model of mergers, but I don’t think a simple linear regression would be it…
A good model of a merger would be a dynamic agent based model in which each airport and each route is modeled as a separate process. We would model consumer choice to not travel, to drive, or to fly. And model the airlines fares on routes dependant on costs to operate and consumer willingness to pay. You’d tune the model until an optimization process produced similar (but not identical) results to the before actuals.
We would then perform the merger putting routes under the control of the new entity, and carry out an optimization to maximize the profit of the new entity… Canceling routes, changing fares, opening up new routes, etc and look at what falls out. Ideally you would have data on before and after previous mergers, and run your dynamic model to optimize profit before and after those mergers and show that you have some predictive validity in those cases.
The structure of that model is mechanistically oriented. People make decisions about travel based on costs and benefits, airlines do too. Airlines cancel routes that don’t have enough passengers, and they add routes where planes are jammed full. They make fare decisions based on competition to try to attract passengers from competitors etc. There are thousands of routes. We should be able to show that we can at least approximately reproduce the choice to operate various routes as part of our model. Just looking at regression models of average price and average occupancy across multiple thousands of routes would be a bad model in all likelihood.
Did you ever write this paper/publish anything about it?
I’m still working on it. Thanks for the encouragement!