Apropos of our recent discussion on the estimation of historical population sizes, Sean Manning writes:
Some archaeologists have measured house sizes for Gini-coefficient-style studies aside from studying human remains to measure nutrition and rates of illness. I think that was what Michael E. Smith meant when he talked about hypothetical data: “archaeology can’t give social scientists population or GDP, but here are some things we can measure that might be useful for social science.”
I asked Manning where the quote came from, and he replied:
I think I got the idea from this response by Smith to a published paper:
This model of inequality in the Aztec Empire is not based on empirical data. While there is nothing wrong with hypothetical models per se, the paper is phrased as if it presents empirical findings. … There are simply not enough data available to do the kind of analysis presented in this paper. The tweaking of data and methods do not produce results that satisfy me as being reasonable estimates of the level of inequality in the Aztec Empire. Perhaps this is just an epistemological difference between our approaches to science and knowledge. Economists might look at this paper as a fine analysis, whereas archaeologists and historians will probably look at it as a study based on hypothetical data, and therefore divorced from the Aztec reality that we study.
Smith has a book that talks about the archaeology of inequality in Aztec Mexico: Timothy A. Kohler and Michael E. Smith, editors, Ten Thousand Years of Inequality: The Archaeology of Wealth Differences (University of Arizona Press, 2019).
Often in social science there is tension between what we can measure and what we would like to know.
You can apply models to the past, but you have to show it can predict the future first. This is done in astronomy all the time for historical eclipses, etc.
Wait – are you dismissing this work unless it can predict future income inequality in the Aztec Empire? If so, then my research is excellent: I predict zero inequality and zero income (with some epsilon below 100% confidence).
The model would need to predict the US, etc values to prove itself THEN you can plug in the guesstimated ancient values.
This seems like extremely basic stuff to me so I dunno how papers like this get through peer review.
Actually, this reminds me of the “billion tons” of copper missing from Michigan: https://en.wikipedia.org/wiki/Copper_mining_in_Michigan
The model of how much copper was mined is completely unvalidated, then the apparent deviation from the estimate is used to generate wild speculations.
Use a simple rule that if there is no demonstrated *predictive* skill (“explaining” historical data is easy and doesn’t count) we can ignore it.
I’m not sure that a model of income inequality in the Aztec Empire would be predictive of income inequality in the US. I take you point, but not all models need to be completely general.
Anoneuoid, Nicholas Taleb has a colourful simile for why this won’t work in Fooled by Randomness: investors (and historians) are like people playing Russian roulette if the rules were constantly changing and could only be discovered by experiment. By the time you have enough evidence to be confident in a theory, the underlying rules will have changed. Eclipses work the same way every time. Societies are dynamic systems and things may never be repeatable. People in the historical sciences often use ethnographic analogies, but the parallels are never exact and the numerical evidence is never as good as we would wish. Really poor rural people tend to be illegible to the state. So most of us don’t try to make general theories or covering laws. One of Michael E. Smith’s terms is mid-range theory.
How does that differ from eg rain falling falling on top of a mountain/hill and predicting where it will eventually end up? Even if you know the laws being followed you still need the terrain to predict the outcome.
Seems more likely the methods used in “soft” fields are not up to the task, then the researchers use “its so complicated” as an excuse.
The exchange between Smith (critique) and the authors is quite thorough and a good example of the kind of exchange I would like to see more often. I find the quote “archaeology can’t give social scientists population or GDP, but here are some things we can measure that might be useful for social science” quite strange, however. It sounds overly defensive, as if population or GDP are more worthy or useful measures than income inequality in the Aztec Empire. I see no need to be defensive about what archeologists study compared to economists or demographers. From what I see, Smith was making a point about this particular research: that while archeologists can provide useful measures, he didn’t find this particular work to be a good example. I have absolutely no background in this area, so I don’t have a comment about the merits either way. What I am commenting on is the predicate in the quote: why preface the comment with a caveat about what archeologists cannot provide, as if the GDP estimates are somehow superior measures in usefulness. But that implicit criterion, I guess Smith would agree with those economists who say that economics is the “queen of the social scientists.” I don’t agree with that – and if someone can point to the source of that sentiment, please do so (because I can’t remember).
A lot of us hoped that the Internet would lead to more extended back-and-forth exchanges between academics in different fields. The rise of Twitter was especially poisonous because the site conventions mixed up partisan gesturing with reasoned professional opinions. It was very hard to trust anything you read there on a controversial topic.
Because quantitative social scientists get more money and media attention, there is a certain amount of social science envy in the historical sciences, and a certain amount of effort to get Senpai to notice us. The problem is that specialists in the last century or so don’t want to wrestle with the things we can actually observe about the past like the size of houses and the amount of fish in people’s diets, they want the same metrics which they get modern bureaucratic states to collect for them. and they tend to be rather naive about the numbers people give them, even though those same bureaucratic states publish tomes on the limits of their GDP or population figures. So the usual result is the specialist in pre-20th-century trying to salvage something from theories built on evidence from the 20th century that don’t work very well or presume we know unknowable things.
Sean:
That’s an interesting point and it motivated me to follow up with a post which should appear in late Jan.
I really don’t get where people are saying there is “no” data. There are data, they are not the same as modern data but they allow measurement and interpretation. In sociology we usually include “unobstrusive measures” like accretion and erosion, and I usually use archeological data as some examples. I would recommend ignoring people who think that if you can’t find measures of a concept that only makes sense in industrial societies with states you can’t learn anything mportant or useful.
Hi elin, in stats terminology, archaeology can often give proxies for the variables that social scientists want, but can rarely show whether that is a good or bad proxy. In the comments above Smith argues that one argument from proxies is not sound. Written sources can give us numbers which could be the result of a careful collection process or could be wild guesses or allusions to numbers in other stories and we can almost never know which. People used to government data often take figures based on proxies or numbers in a chronicle as approximately correct within say 10%, rather than as one end of an order of magnitude of possibilities (and it is not hard to find estimates of the population of a New World region in 1492 that differ by a factor of 5 or 10 or 20). I think that the difference between a guess or an estimate based on proxies and an observation is important.
What’s also interesting to me about that quote is that archaeologists are (often not always) considered a subdiscipline of anthropology, and there very much are demographers in anthropology, some of whom are trying to figure out population and population dynamics of historical societies
Well yes, that applies right across the board in science although perhaps in social science it’s more prevalent than in other sciences where the available information may be more abundant and better-defined. A related example to the Aztec inequality question and its information deficit is the nature of the archaic organisms at the root of our evolutionary lineage – e.g. the Last Universal Common Ancestor, Last Eukaryotic Common Ancestor, archaic species from which the vertebrate lineage derived and so on. We can increasingly winkle out information about the gene complement of these species from existing sister species and extrapolating from common gene complements in wide ranging vertebrate species to derive what is expected to have existed when various evolutionary lines diverged. That’s very useful in attempts to reconstruct where genes and evolving physiologies came from.
Not surprisingly this isn’t an exact science but no one gets their knickers in a twist if the analyses have considerable uncertainties. Any analysis with its interpretations can provide hypotheses that create context for directing further effort and generally we tend to move closer either towards (i) the reality (we need some criteria to assess this) or (ii) a consistent provisional description that may be useful. IMO it’s helpful in such studies (like the Aztec one) to recognise that the interpretation is a hypothesis and not a conclusion.
I’m reminded of those human archaeology popular science books where an entire edifice is constructed around the discovery of a hominid footprint or a femur. One may as well be relaxed about this stuff. Of course an expert in the topic (like Michael Smith in the Aztec example) may find themselves incandescant in the face of what they consider a deficient analysis but that’s also OK since it gives them the chance to display their expertise and initiate some debate. The discussion between Smith and Alfani (author of the Aztec Gini coefficient paper he’s criticising) on PubPeer is interesting, and as is often the case one (as a neophyte) can learn more about a topic from this sort of exchange than from reading a paper.
Hypothesis: A = .73
Data: B=.73
Important Handwaving Paragraphs: We expect A to be like B, or related to B, or comove with B or have some meaningful relationship to B. Plus, there are no data on A.
Conclusion: QED
Honestly this isn’t the worst argument. Problems develop though, when (a) someone claps back at the handwaving paragraphs; or (b) the argument that B=.73 is pretty shaky as well.
Somewhat along the lines of this post – and quite in line with many posts over the years – is this essay from today’s Chronicle of Higher Education: https://www.chronicle.com/article/when-bad-research-feels-good. Nothing new, but a good description of the tensions academics can feel in some disciplines and the ease with which personal beliefs bias research practice. The author focuses on social psychology and anthropology, but I’m not convinced those fields are any different from all the others.