This is Jessica. Recently I got asked, on a podcast, what famous scientific figure alive or dead, real or fictional I would have dinner with, and for the sake of choosing someone who was both brilliant and sounds like an inspiring person to be around, I said David Blackwell.
I didn’t really know Blackwell’s work until maybe a year or year and a half ago, when I was introduced to Blackwell Ordering by a colleague. It’s a way of quantifying the instrumental value of some information generating process (aka channel or experiment or information structure) before the state of the world realizes. Imagine we have two different forecasts for predicting who will win an election, based on some set of signals like poll results, and we can choose which one to consult to make some decision, like how much of our time or money we will allocate to help our preferred candidate prior to an election. How do we decide, given that the value we get out of the forecast (assigned by some real valued payoff function) will depend on the decision we make and the state that ultimately realizes? If we know the probability distribution over the possible signals for each possible state of the world, then Blackwell’s theorem tells us that under certain conditions we can choose which forecast to use even without knowing the specific utility function or the distribution over inputs that we are dealing with. The constraint that needs to be satisfied is that one information structure needs to be a “garbling” of the first (or in Blackwell’s terms, the first needs to be sufficient for the second), meaning that we can represent the second as what we get when we apply some post-processing operation to the first (where both are represented in matrix form).
Once we establish that one forecast is sufficient for the other, we know the second one can’t give us more information about the state: Blackwell showed that the expected utility of deciding using the optimal strategy for the original (non-garbled) forecast is always at least as big as using the optimal strategy with the garbled forecast. This is not just for one particular instantiation of a decision problem, but for any input distribution or utility function. So, the ordering gives a concise summary of the statistical conditions under which one experiment is more informative than another. I’m not sure the extent to which work like Blackwell’s, which he did in the early 1950’s, directly influenced later learning theories that are similarly agnostic to the input distribution, like PAC learning, but this way of thinking seems ahead of its time. When I first learned about it, I found Blackwell ordering exciting because I was frustrated with how dependent our knowledge of what makes a better visual representation of uncertainty tends to be on specific decision setups that researchers study. So the idea of theory for doing unconditional analysis of information structures reflecting uncertainty around some outcome was like a missing piece I was looking for. Since then I’ve been paying more attention to other work on the value of information, such as in economics.
More recently I learned more about some of Blackwell’s other major contributions, like approachability. The original formulation asked under what conditions we can expect the row player in a minimax game with vector payoffs to be able to get his payoffs to approach some target set, making it a generalization of minimax for vector payoffs. But there are connections to online learning and forecasting; it turns out Blackwell’s result implies no-regret learning and calibration. There’s also Rao-Blackwell theorem, which says that (at least with enough data) it’s possible to make a better estimator of your parameter by taking the conditional expected value of your estimator given some sufficient statistic for that parameter.
This fall I attended a workshop on data-driven decision processes, where there was a day devoted to Blackwell. Some of these were by people who spent time around him, and shared anecdotes about what he was like to work with. He sounds like an inspiring person to be around, someone who undoubtedly experienced a lot more pushback along the way that most of his faculty peers but continued to make major contributions and positively affect the people around him in all sorts of ways. Someone mentioned for instance how he had high standards for communicating ideas in the most direct, comprehensible way, like when teaching, which I respect. There are talks listed under the last day here that cover his work and influence for those that are interested.
However, there is one anecdote that I keep coming across as I’ve been finding out more about Blackwell– not about Blackwell himself but about his experience as a Black statistician–that I have to complain about. It involves Blackwell’s first interactions with the UC Berkeley math department, which then contained statistics, where he later spent years as a professor and chair. Apparently, he was considered as a hire in 1942 but was not extended an offer, because of racism. Many accounts describe how the offer was blocked by the spouse of a faculty member, e.g., “Blackwell would later in his career find out the then head of the math department’s wife protested Blackwell’s hiring. It was customary to host faculty members in their home and the wife objected to hosting a black man in her house.”, or “Blackwell was blocked by one of the ‘faculty wives.’” Or, put differently: “In 1954, after an initial attempt in 1942, which failed due to the racial prejudice of some faculty families, Blackwell was appointed Professor.”
To be clear, I do not doubt that someone’s wife (or more specifically, the chair’s wife as some records describe) protested to Blackwell getting hired. Racism is not hard to find even now, so pre-civil rights era it does not seem surprising that some professor’s wife would have made blatantly racist comments. And I’m glad that the bias that Blackwell had to deal with during his life is brought to light in recounting his career.
But… isn’t it a little odd, so long after the fact, to be talking about someone’s wife as the reason Blackwell wasn’t offered a job in what was presumably at the time an all or mostly male department? I respect that speakers and authors who retell this want to acknowledge the racism Blackwell experienced in the mostly white academic institution, and I can understand why some of the original faculty involved might have been frustrated by the influence of someone’s wife. But eighty years later, it seems kind of weird to hear this retold, as if we’re going out of our way to put the blame on some woman who wasn’t even part of the department. The optics distract from the more interesting story of who Blackwell was.
I doubt the reasons for bringing up this particular anecdote are to intentionally redirect blame, so much as people have heard it and decide to echo it to tell a more entertaining story. From looking around online it seems like it originated from a biography on Neyman written by Constance Reid.
If all of the professors were OK with hiring a black man, but one of the wives objected and convinced their professor-spouse to object, then maybe it’s OK to emphasize the role of the wife. Framing it like this seems OK to me: “Blackwell would later in his career find out the then head of the math department’s wife protested Blackwell’s hiring. It was customary to host faculty members in their home and the wife objected to hosting a black man in her house.”
But this seems wrong: “Blackwell was blocked by one of the ‘faculty wives.’” Unless the ‘faculty wife’ had a vote, this is just wrong. A wife might successfully lobby her husband to block the guy, but it’s the husband doing the blocking.
Additionally, I’m suspicious of the story. I can easily imagine a professor saying “Oh, you know, we all wanted to hire you several years ago but Professor Dickwad’s wife objected.” Especially if Professor Dickwad had retired by then. Not that I’m doubting the wife was racist, lord knows that’s believable enough, but it’s also believable that there were other people in the department who didn’t want to hire him either and that they seized on Professor Dickwad’s wife’s objection as a reason. This reminds me of Errol Morris’s investigations into the Crimea cannonball photos, and his other discussions of the difficulties of determining what really happened in historical events.
Phil:
An interesting twist to all this is that it seems that Errol Morris got things wrong with that “Fatal Vision” killer. It seems similar to the Bill James situation: Morris got his reputation by going against conventional wisdom and being correct, and it seems that led him to be overconfident and too dismissive of other perspectives. It’s a kind of Peter Principle of contrarian pundits.
Phil said,
“If all of the professors were OK with hiring a black man, but one of the wives objected and convinced their professor-spouse to object, then maybe it’s OK to emphasize the role of the wife. Framing it like this seems OK to me: “Blackwell would later in his career find out the then head of the math department’s wife protested Blackwell’s hiring. It was customary to host faculty members in their home and the wife objected to hosting a black man in her house.”
This brings to mind an incident involving my mother, who taught children’s literature. Once she was giving a continuing education workshop on that subject for elementary school teachers, in a city she was not familiar with. On the first day, she noticed that the audience was sitting in segregated style — white teachers on one side of the room, and black teachers on the other side of the room. This bothered her, so she quickly started a “get acquainted” activity that was intended to get the seating arrangement unsegregated.
Jessica:
Thanks for that. Here are my thoughts on Blackwell from a few years ago. You might be interested in the bit where he said, “I’m not interested in doing research and I never have been, I’m interested in understanding, which is quite a different thing.” It seems to me that this statement represents a way that people used to think that we’re moving away from. In Blackwell’s world, “understanding” was not “research.” But, to us, developing tools for understanding is an important part of research. I think this is part of a larger cultural shift, or maybe it’s just a difference between mathematics (Blackwell’s world) and engineering (our world).
Oh, thanks, didn’t realize you had a post on him!
I had heard the line about research vs understanding elsewhere recently too. I’m not sure I could have related to it a few years ago, when I was very gung-ho about devising new tools or frameworks. But now, being in a phase where I’m feeling unable to claim to have better approaches until I more deeply understand why certain intuitions I’ve had about problems seem wrong, I can understand why someone might protest a focus on research over understanding. I do think there’s a higher value put on understanding (even in the absence of any practical implication) among theorists (in stats, CS, econ etc) compared to the rest of those fields.
For what it’s worth, I believe the story about Blackwell’s 1942 non-hiring did not start with Reid’s book. There are interviews with Blackwell himself who tells the story that way. I can totally see why that framing sounds odd now, 80 years later, as you say. But it is how Blackwell himself told it.
Here’s a YouTube clip: https://www.youtube.com/watch?v=Mqpf9tw44Xw
Starting around 1:54:
“Neyman had decided to appoint me, and recommended it to the mathematics department here in 1942, and the mathematics department approved. But the head of the mathematics department went home and mentioned this to his wife. And she drew the line. As I understand it, her words were that she was not going to have any darky in her house…”
A similar version is in “An oral history with David Blackwell : oral history transcript” from 2003, at Berkeley’s Bancroft Library / University Archives:
https://digitalassets.lib.berkeley.edu/rohoia/ucb/text/blackwelldavidor00oralrich.pdf
See p.107, which is the 127th page of the PDF.
That said, thank you for bringing up Blackwell’s incredible contributions and reminding us not to focus just on this one story!
Of course that raises the question: If the story comes from Blackwell himself, why did *he* frame it that way?
I can imagine at least one reason why Blackwell himself might go “out of [his] way to put the blame on some woman who wasn’t even part of the department.” By the time he did these interviews, he *had* been hired at Berkeley and had worked with some of those same colleagues for years.
So perhaps he himself didn’t want to start a public feud after years of eventually working with some of the same colleagues who hadn’t hired him in ’42. Surely he knew the chair’s-wife story was a (weak!) excuse, but perhaps he tolerated it to let his colleagues save face — an imperfect solution that let him keep the peace with the other faculty members he was stuck working with.
In any case, you’re right that we don’t necessarily have to keep telling the story this way.
For all we know, when he was among friends he might have ended the story with
“…and I can’t believe those cowards in the math department didn’t overrule her”
or
“…and I can’t believe those cowards threw her under the bus as a fake excuse instead of just admitting they were too prejudiced to hire me.”
Interesting, didn’t realize he told it that way. But yeah, seems like telling it this way probably reduced awkwardness between him and his colleagues.
I was fortunate enough to take an undergraduate class (Stochastic Processes) from Prof Blackwell at Berkeley in 1977 or 78. I felt at the time that he was one of the best teachers I’d ever taken a class from. Doing the work was easy and fun, at least partly because of him. A couple of anecdotes:
1) He was endearingly absentminded about some things, like chalk dust. He always carried pieces of chalk in his coat pockets—he always wore a coat and tie, and dressed well, but by the end of the day his coat would have shrouds of chalk dust about the pockets.
2) I learned a great deal from him and made sure to discuss things with him a few times in office hours, which he seemed to enjoy. I fancied I was his best student. Then he had to miss a lecture and he asked another student in the class to give it in his place. I was crestfallen that he didn’t ask me, so talked with the guy (politely, I wasn’t annoyed, just curious) and asked why he thought Blackwell had chosen him. He said that before the class started he’d gone through the textbook and done all the problems and talked about it with the prof. Well who could argue with that? ;-) (That’s a synopsis of my experience at Cal, I never stood out because there were always folks like that. Mr A- was how I came to see myself.)
3) As well as being a fantastic teacher he was a super nice man (maybe not independent variables) with a good sense of humor. I wish I’d discovered him earlier so I could have taken more classes with him.
Oh hey, this is Blackwell of Rao-Blackwellization? Of particular importance to Stan as well! I had no idea he was a Berkeley faculty member. Being turned down from UC Berkeley, he’s in good company
If you studied dynamic programming and Markov Decision Processes back when I did, Blackwell was well known and influential. I haven’t worked in that area for a long time so I have no idea if any of his work in these areas are still influential.
So, here is a question. I read that Blackwell wrote a textbook on Bayesian statistics, but I think I remember reading that Andrew got in trouble with the Berkeley statistics dept. for writing a book on Bayesian statistics. It seems like there is a story there.
John:
1. Blackwell’s work on Bayesian statistics was much more theoretical than mine. I arrived at Cal around the time that Blackwell retired, and I taught the undergraduate Bayesian inference course that he’d taught. As I recall, his version of the class covered things such as conjugate priors that were pretty distant from applied Bayesian statistics as I saw it, and I couldn’t really figure out how to teach a Bayesian statistics class that I’d like at an undergraduate level, so I changed it into a decision analysis class. I taught that course 2 or 3 times at Cal and another 2 or 3 times at Columbia and I really liked it—it seems to me like a perfect topic for an undergraduate course, with a combination of math and social science applications. I did not use Blackwell’s book but I tried to teach it in a Blackwell spirit of applied math. I stopped teaching the decision analysis class a couple decades ago because there was a demand for courses in some more urgent applied topics such as survey analysis, Bayesian inference, and some other things, but maybe I’ll return to it sometime.
2. I did teach a course on Bayesian statistics at Berkeley but they didn’t want me too. When I was hired they told me that one of the courses I’d teach could be on whatever I wanted, and I said I wanted to teach a Ph.D.-level course on applied Bayesian statistics, something that was not on their curriculum at all (other than that undergraduate class that Blackwell had formally taught). At first they wouldn’t let me do it but then I showed them the letter where they said I could teach what I wanted, so they relented. I don’t remember all the details, but I think I taught it under course number for Multivariate Statistics or something like that. A couple years after that one of the Ph.D. students I was advising told me that the reason he wanted to work with me is that one of the other faculty had told him not to take my course. This might have been a different person than the tenured statistics professor who came up to me after a seminar I’d given and told me that he’d not before realized how trivial all that political science stuff was. I guess he’d been too busy scoping out the female postdocs to pay attention to the content of my research.
3. I think my colleagues were ok with Bayesian statistics in theory but not in practice. If my book had been a bunch of theorems, maybe they would’ve been OK with it.
4. Blackwell himself was positive about my work and wanted to promote me, but he was not interested in getting into a fight with the other faculty members about it. (He was retired at the time of my promotion but I think the retired professors got to vote too. I’m not sure about this, though.)
On #2 – wow. I hope he at least said ‘This is trivial’ in a friendly sort of way.
It reminds me of another quote attributed to Blackwell: “Some people will help you, and some won’t, and it’s fair.”
No, it was not in a friendly way. It was not exactly hostile, either. It was more of an aggressive mathematician style, where he just assumed he was correct, so friendly/hostile didn’t matter to him. The funny thing is that he did lots of work on genetics, which is pretty trivial compared to working with survey responses!
E. T. Jaynes mentions Blackwell one time. After giving talks explaining that the MAXENT procedure was generally useful (before the Burg algorithm in spectral analysis made that more concretely obvious) and not meeting much success:
“Attempts to explain this to other physicists, engineers, mathematicians, and statisticians at Stanford and Berkeley, and to the Army Engineers, met with strictly zero success–with only one exception.
David Blackwell saw the point at once, and put his finger on the basic ‘fairness’ property of the MAXENT algorithm: It doesn’t not allow you to assign zero probability to any situation unless your information really rules out that situation.”
Interestingly, the stat procedures seemingly favored by Berkeley, such as hypothesis testing, wind up in effect assigning zero probability to situations the information hasn’t ruled out.