“Smell the Data”

Mike Maltz writes the following on ethnography and statistics:

I got interested in ethnographic studies because of a concern for people analyzing data without an understanding of its origins and the way it was collected. An ethnographer collects stories, and too many statisticians disparage them, calling them “anecdotes” instead of real data. But stories are important; although they only give you single data points, if you have a number of them you can see how different they seem to be. That helps you determine if there is more than one process generating the data. in fact, I’ve noted that the New York Times now often augments an article with a bunch of stories about how different people were affected by the focus of the article.

Here’s the way I [Maltz] described the benefit of ethnography in the introduction to a book (Doing Ethnography in Criminology, Springer 2018) that Steve Rice and I edited:

I’m best known for quantitative, not qualitative, research. An engineer by training, I had never taken any courses in social science when I began teaching in a criminal justice program in 1972. My entire criminal justice experience up to that point was based on my having been a staff member of the National Institute of Justice from 1969 to 1972, and I was hired by NIJ because of my engineering background and my experience in police communications—see below.

My introduction to social science and to the research techniques that were then used by its practitioners began when I joined the criminal justice faculty of the University of Illinois at Chicago. I was put to work teaching social science statistics and, not knowing much about it, used the books that others used before me. But even then I was mystified by the common practice of looking to achieve a low p-value as the be-all and end-all of such research. Untenured and with no experience in the field, I taught what others thought was important. But I soon wised up, and described my concern about the methods used, some 30 years ago (Maltz, 1984, p. 3):

“When I was an undergraduate in engineering school there was a saying: An engineer measures it with a micrometer, marks it with a piece of chalk, and cuts it with an axe. This expression described the imbalance in precision one sometimes sees in engineering projects. A similar phenomenon holds true for social scientists, although the imbalance is in the opposite direction. It sometimes seems that a social scientist measures it with a series of ambiguous questions, marks it with a bunch of inconsistent coders, and cuts it to within three decimal places. Some balance in precision is needed, from the initial measurement process to the final preparation of results.”

And I further expressed my concern about the focus on “statistical significance” in a subsequent article (Maltz, 1994). Ethnography is a welcome and much-needed departure from that type of research. It deals with individual and group behavior that doesn’t conform to statistical or spreadsheet analysis. Yes, an ethnography may just be a single data point, but it often serves as a marker of importance, an exploration of what additional factors should be considered beyond the usual statistics, or as a counterexample to some of the more positivist studies.

In this regard, three examples provide additional context to my strong belief in the need for a qualitative orientation. The first was my initial experience while consulting on police communication systems (true electrical engineering!) for the Boston Police Department from 1966 to 1969. To satisfy my curiosity about the ways of the police, I requested, and was granted, permission to conduct an “experiment” on police patrol. The number of patrol cars in one police district was doubled for a few weeks to see if it had any effect on crime. And it did: compared to the “control” district, which had no arrests, the “experimental” district had six arrests. Moreover, there were no arrests at all for the same time period in either district in the previous area, so I could calculate that p = 0.016, much less than 0.05. What a finding! Police patrol really works!

On debriefing one of the arresting officers, one of the first lessons I learned was that police officers are not fungible. There are no extra police officers hanging around the station that can be assigned to the experimental district: they have to be drawn from somewhere else. The additional officers, who made all of the arrests, were from the BPD’s Tactical Patrol Force—the Marines of the department—who were normally assigned to deal with known trouble spots, and the two districts selected for the study were generally low-crime areas.

In fact, the TPF officers already knew that a gang of car thieves/strippers was active in the experimental district and decided to take them out, which resulted in all of the arrests they made. They couldn’t wait to get back to working citywide, going after real crime, but took the opportunity to clean up what they considered to be a minor problem. So after that experience, I realized that you have to get under the numbers to see how they are generated or, as I used to explain to students, to “smell” the data.

Another example: Some years ago I was asked to be an expert (plaintiff’s) witness in a case in the Chicago suburbs, in which the defendant suburb’s police department was accused of targeting Latino drivers for DUI arrests to fill their arrest quotas. My job was to look at the statistical evidence prepared by another statistician (the suburb’s expert witness) and evaluate its merits. I was able to show that there were no merits to the analysis (the data set was hopelessly corrupted), and the case was settled before I had a chance to testify.

What struck me after the settlement, however, was the geography and timing of the arrests. Most of them occurred on weekend nights on the road between the bars where most of the Latinos went to drink and the areas where they lived. None were located on the roads near the Elks or Lions clubs, where the “good people” bent their elbows.

I blame myself on not seeing this immediately, but it helped me to see the necessity in going beyond the given data and looking for other clues and cues that motivate those actions that are officially recorded. While it may not be as necessary in some fields of study, in criminology it certainly is.

A third example was actually experienced by my wife, who carried out a long-term ethnographic study of Mexican families in Chicago (Farr, 2006) all of whom came from a small village in Michoacán, Mexico. Numerous studies, primarily based on surveys, had concluded that these people were by and large not literate. One Saturday morning in the early 1990s, she was in one of their homes when various children began to arrive, along with two high school students. One of the students then announced (in Spanish, of course), “Ok, let’s get to work on the doctrina (catechism),” and slid open the doors on the side of the coffee table, revealing workbooks and pencils, which she distributed to the kids.

On another occasion, my wife was drinking coffee in the kitchen when all of the women (mothers and daughters) suddenly gathered at the entrance to the kitchen as someone arrived with a plastic supermarket bag full of something—which turned out to be religious books (in Spanish) on topics such as Getting Engaged and After the Children Come Along. Each woman eagerly picked out a book, and one of them said, “I am going to read this with my daughter.”

Clearly these instances indicate that children in the catechism class and the women in the kitchen were literate. The then-current questionnaires that evaluated literacy practices, however, asked questions such as “Do you subscribe to a newspaper? Do you have a library card? Do you have to read material at work?” In other words, the questionnaires (rightly so) didn’t just ask people outright “Can you read?” but rather focused on the domains they thought required reading. Yet no questions dealt with religious literacy, since literacy researchers at the time did not include a focus on religion. The result? The literacy practices of these families were “invisible” to research.

These anecdotes are but three among many that turned me off the then-current methods of learning about social activity, in these cases via (unexamined) data and (impersonal) questionnaires. Perhaps this has to do with my engineering (rather than scientific) background, since engineers deal with reality and scientists propound theories. To translate to the current topic, it conditioned me to take into consideration the social context, a recognition that context matters and that not all attributes of a situation or person can be seen as quantifiable “variables.” This means, for example, that a crime should be characterized by more than just victim characteristics, offender characteristics, time of day, etc. and that an individual should be characterized by more than just age, race, ethnicity, education, etc. or “so-so” (same-old, same-old) statistics. These require a deeper understanding of the situation, which ethnography is best suited, albeit imperfectly, to do—to put oneself in the position, the mindset, of the persons whose actions are under study.

12 thoughts on ““Smell the Data”

  1. This reminds me of the common experience that if you actually know about the substance of a newspaper story, you so often find it to have gotten things wrong in some fashion.

    • Agreed. In fact, I used to ask my students if they had first-hand knowledge of anything reported in the media. When I asked the follow-up question, “How accurately was it described?”, most of the time they said that it was not accurate at all.

  2. As someone with a background in social sciences (economics, sociology) trying to do statistics, I find a strong contrast between the richness and variability given in ethnographic descriptions of people’s behavior (and the reasons people give for their behavior) and the roughness with which statisticians and quantitative researchers (including myself) sometimes just go over, kind of ignoring to a large extent the variability and heterogeneity in a population, and summing it all up into one single number, within some interval to capture uncertainty. I wonder whether heterogenous treatment effects should be the default setting in doing social science, and I wonder too whether the discussions around personalized medicine are speaking to this variability. Do you know anyone working in Bayesian statistics for individual treatment effects/heterogeneous effects? I could use some guidance here…
    Thanks for posting this!

  3. Thanks for sharing this, it’s a good corrective to the “scientism” that seems omnipresent in many social sciences. By scientism, I mean specifically the idea that by dealing with quantitative data, any results are necessarily more “true”. Truth lies not in the data alone, but in the connection between the data and the latent constructs out in the world. Ethnography is an important tool for establishing that connection.

    My only gripe is with the phrase, “engineers deal with reality and scientists propound theories.” I think both professions are about dealing with reality, just in different ways, though in practice both can fail to do so effectively. As Maltz himself notes, engineers often fail to “deal with reality” on its own terms by being absurdly specific while planning and then basically “going with their gut” in execution.

    And it is true, of course, that part of the job of scientists is to produces theories, but good theories are ones that have clear connections between their constructs and reality. Psychology and social science are areas where that connection often breaks down.
    Modern theoretical physics is replete with theories that make only token efforts to connect to anything measurable or predictable.

    So while I think Maltz’s swipe at scientists as “propounding theories” without “deal[ing] with reality” mischaracterizes what these professions are *supposed* to do, it is accurate as a criticism of how things are in many fields, and at least in social science, ethnography seems an important corrective.

  4. Ethnography has become relatively common in research on interfaces in computer science, though debates about how it produces knowledge relative to quantitative work continue to occur (some areas, like visualization/statistical grpahics, have been heavily steeped in logical positivism so its still hard for some people to understand the value of it). The paper on people in rural communities in Penn interpreting graphs struck me as more ethnographic https://statmodeling.stat.columbia.edu/2019/05/31/data-is-personal-and-the-maturing-of-the-literature-on-statistical-graphics/
    There was also this paper a few years ago by Miriah Meyer and Jason Dykes that discussed how “design studies” where researchers get involved in some domain specific problem and build software for it without necessarily designing controlled evaluations, can be rigorous: https://arxiv.org/pdf/1907.08495.pdf

  5. I’ve commented on this before, so I won’t belabor the point, but I *strongly* agree with this post. A single case, deeply (and personally) investigated can reveal context, causal mechanisms, and potential pitfalls that someone who looks only at quantitative sample data would never know. Representativeness is not everything.

    The only data set I ever created from scratch was the product of multiple on-site visits and open-ended conversations with people directly involved with what I was studying, and I’m sure the end product was more informative and reliable because of it.

  6. Once you start really looking into how the data sausage is made (or go through the grueling process of making some on your own), you quickly realize that ‘there ain’t no such thing as raw data.’
    Careful examination of how the data is generated can be as informative as the resulting data itself.
    Maybe people don’t talk about this as much because it is so trivial, but I doubt that is the main reason.

  7. Fred said,
    “Maybe people don’t talk about this as much because it is so trivial”

    I would say, “Maybe people don’t talk about this as much because they think of it as trivial”

  8. The three examples given highlight to me, primarily, the costs of not approaching these questions as a scientist, in terms of research design, theory, and measurement. They also highlight benefits ethnographic approaches, but only as a post hoc fix for studies implemented without sufficient attention to research design, theory, and measurement.

    Exhibit A: The crime study had an n of 2 and a terrible operationalization of “crime” (number of arrests). These problems, and their potential consequences, could’ve been anticipated prior to the study by someone well versed in research design methods.

    Exhibit B: The reason he didn’t see the pattern in DUI arrests is that he approached the data without a theory (his own or, preferably, one derived from the relevant literature). If the lesson he learned here is to pay closer attention, that’s the wrong lesson. He should’ve learned that no one could possibly anticipate all potential analyses and their relevance, a priori, in the absence of theory.

    Exhibit C: That the literacy surveys of the time were not valid for use with this population would’ve been apparent from their psychometrics–I’m sure assessments of reliability and validity would’ve come from a very different population, if they were assessed at all. Precisely which items had differential item functioning (DIF) could’ve been determined with a conceptual map of the construct.

Leave a Reply

Your email address will not be published. Required fields are marked *