Rules of historical evidence

Our Columbia colleague Andy Nathan recently wrote a review of a recent biography of Mao. My interest here lies not so much in the subject matter (important though it is) but rather in Nathan’s comments about historical research methods, in particular, for the key issue of what can or should be believed.

One of our goals in creating the Quantitative Methods in Social Science program was to give students an overview of research methods in a broad range of social sciences: history, economics, sociology, political science, and psychology. We found that the students were eager to learn more about statistical data analysis (regression, scatterplots, chi-squared tests, time-series analysis, Markov models, and so forth), were somewhat interested in the experimental methods used in psychology, but showed almost no interest in the sources of historical data. (They enjoyed the lectures of Herb Klein, the professor who taught the “history” section of the course, but had no interest in working themselves with original documents.)

Andy Nathan’s review is interesting in that it focuses on methodological issues, in the context of a topic of historical intersest. The book under review (Mao: The Unknown Story, by Jung Chang and Jon Halliday) is the product of twenty years of research including hundreds of interveiws and thousands of documents in several languages. Nathan points out, however, that much of the information they present is hard to confirm:

The inaccessible sources are of two kinds: anonymous interviews and unpublished documents or books. The former include ‘the wife of a Shanghai delegate’, ‘interview with a local Party historian’, ‘interview with an old underground worker’, ‘interviews with people who had been told’, ‘interview with a staff member who knew about Mao’s account’, ‘interviews with Mao’s girlfriends’, ‘interviews with Mao’s personal staff’, ‘interview with a Russian insider’ . . .

Basing their argument on such sources, Chang and Halliday claim that the most famous battle of the Long March, at the Dadu Bridge in 1935, never took place. Their key piece of evidence is an interview with a ‘sprightly . . . local woman . . . who was 93 years old when we met her in 1997’, supplemented by an interview in 1983 with the then curator of the museum at the bridge. Their related claim that Chiang Kai-shek had deliberately ‘left the passage open for the Reds’ is unsourced.

Chang and Halliday state that Mao’s chief political rival in Yan’an, Wang Ming, was poisoned by a Dr Jin, acting at Mao’s behest. They say that this was established by an official inquiry, whose ‘findings, which we obtained, remain a well-kept secret’. They cite the document in the notes, but do not say where it can be seen. They assert that Mao blamed the Indonesian Communist Party for failing to seize power in Jakarta in 1965. Their evidence is a conversation Mao had with Japanese Communists in 1966, in particular some remarks which, according to the source note, ‘were withheld from the published version’ of the talks and ‘were made available to us by the Japanese Communist Party Central Committee’. How other scholars can consult these remarks isn’t stated.

Their source [for a different claim] is a three-volume work called ‘Documents for Researching the Cultural Revolution’ compiled by the People’s Liberation Army Defence University, which they describe as unpublished. They do not say where they saw it.

What’s interesting about this example for me (and perhaps will make it uninteresting to others) is that there is essentially no disagreement between Nathan and Chang/Halliday on the big picture: they all agree that Mao was brutal and devious in attaining and maintaining power and in achieving his political goals. They also agree that historical research is difficult in a totalitarian setting. Nathan writes,

Of course, anonymous interviews and unpublished sources are often used in reputable China scholarship. They have to be, because of the secrecy imposed by the regime on its own history and workings. I [Nathan] have engaged in such research myself.

He then continues,

What is troubling about Mao: The Untold Story is the authors’ failure to give readers any information to help them to evaluate their sources’ reliability. . . . How was it possible to gain access? Who gave authorisation or protection, formal or informal, to this project, or if none was given, how was secrecy maintained as the research progressed? How were the interviewees found? In what settings were they interviewed? In what manner were they questioned? How were records of the interviews kept? What motivations did informants have for talking? What methods were used to confirm their identities and to corroborate their information? How were unpublished sources obtained? How were they authenticated? Where, if anywhere, may they be consulted by other scholars (and if they can’t, why not)?

Nathan then goes on to mention several examples of claims in the book that he does not see backed up by evidence.

As a social scientist who usually uses data collected by others, I’m interested in seeing these discussions because they give me a sense of the rules that scholars can use to come to agreement about the information in historical sources.

3 thoughts on “Rules of historical evidence

  1. I have spent most of this week reading the cvs and job paper of freshly minted econ PhDs. I was quite interested in one of the job papers to see statistics quoted from Wikipedia. While the data was innocuous enough that the anonymous author probably got it right, it seems an odd scholarly source, especially since he didn't cite the day he saw this on Wikipedia, and it could be different the next day. I remarked to a colleague that sourcing a fact to Wikipedia is not that far from sourcing it to "a guy I met in a bar the other night."

  2. Jim,

    My point was that we like results to be replicable. In experimental science (or social science), that means that the experiment can literally be replicated. In observational studies, it means the effect will show up when new data are analyzed; also it means that the actual data analysis can be replicated. In history, I guess it means that, as much as possible, inference be based on openly-available documents or information that can be double-checked. I hadn't thought much about this until I read Nathan's article.

    P.S. The MIT site is amusing. Although there the point is that any human can quickly tell that the generated papers are gobbledygook–they're just making the point that the profit-making organizers of the fake conference don't care. So this is slightly different from Nathan's comment on the history book.

    I do feel a little sad about the speakers at these fake conferences, though. I'll post my story on this next week.

Comments are closed.