Here’s an interesting and informative rant I received recently in the email:
This document is a consultant’s report to the Traverse City Convention & Visitor’s Bureau, quoted — literally photocopied into — a market analysis for an application for an approx. 270,000 square foot shopping center. The full report is here. On page 6 of the .pdf, we are told the following:
“After extensive evaluation and testing of these variables [that possibly determine tourist visitor volume to Grand Traverse County] for their predictive ability, the Consultant determined there are three variables with statistically significant associations. These are population in Grand Traverse County, Gross Domestic Product (GDP), and the External Event dummy variable.
“The Consultant found GDP [national, not regional or local] alone is a significant predictor however [sic] it does not hold up in association with either Grand Traverse Population or the External Event dummy variable.”
The Consultant then goes on to run a regression using GT population and the dummy, but not GDP. The resulting equation has an adjusted R-square of .95, and F=87.0. While GT pop has a t-value=10.9 & p=.000012, the dummy isn’t significant (p=0.3). The Consultant thus takes GT population projections out to 2025 to forecast annual tourist visits for that time frame.
That seems rather sketchy to me. Correct me because I’m likely wrong, but the Consultant basically said that 95% of the variation in annual tourist visits was due to (predicted by) county population, and then used population projections to forecast future tourist visits. And even though GDP was a significant variable, she used population instead, with no explanation why. (Or, none that I can find.) GDP and population were apparently the only two significant variables (though we don’t know how population held up if she removed the insignificant dummy from the specification) of the host of variables she tested; e.g., DoD/military contracts, even though our military presence is limited to a couple Coast Guard helicopters. (And her regression is based on about 10 data points.)
Surely, local population can’t be the driver of tourist visits. It does seem reasonable that population is driven by tourism, since people who visit here might end up wanting to move here, no? That seems to be a questionable variable for trying to forecast tourism in the future, when at least one other significant variable, GDP, is available — even if that was found by data mining as well.
I wish I could say this is typical, but in my experience, local units of government, &c., pay money for analyses even more questionable than what I just presented. For example, the market study in which the above was quoted reports consumer demand in 2005 $194,896,255 less than supply. Setting aside the problems this claim has in view of economic theory, the values labeled “demand” and “supply” are consumer expenditures and retail sales: retailers sold approx. $195 million more that consumers purchased. And there is no explanation of why this is; in 2005, within a 50-mile radius, consumers spent $1,371,392 on “News Dealers and Newsstands,” while retail sales in the same category was $0, and there is no explanation of that $1.4-milion gap!
Well, I [my correspondent] guess there’s no real point to this email other than to complain, and shouting at the sky is getting me a lot of strange looks. I’ll close by just asking you to ask your students to get involved in their communities, and at the very least, act as bullshit detectors and raise their voices when something smells.
This certainly doesn’t surprise me: I’ve seen worse from paid statistical consultants on court cases, including one from a consultant (nobody I’ve ever met or know personally in any way) who reportedly was paid hundreds of thousands of dollars for his services.
The key problems seem to be:
1. Statistics is hard, and not many people know how to do it.
2. The people who need statistical analysis don’t always know where to look.