This is Jessica. I’m back in Chicago now, but I spent the fall visiting an institute for theoretical computer science at UC Berkeley. I’ve been gravitating toward theory at the CS/stat/econ intersection in the interest of being better able to characterize or predict the value of a more informative data summary for a decision problem. The topic of the program I attended was sequential decision making, which covered more classic CS/operations topics like reinforcement learning, multi-arm bandits, competitive analysis of algorithms, etc. as well as some econ-focused approaches like information design. The goal in the CS subset is often optimal or approximately optimal algorithms given some objective, whereas in econ it tends to be simplified characterizations of a solution that nonetheless provide some new insight into what’s going on.
What I want to write about is a certain tension that becomes palpable for me when I try to take theory more seriously, between the spirit of “doing theory” and the need to have some practical example to motivate the theory.
When it comes to motivating applications, some theory papers get away with saying little to establish their relevance, because they start with some well established problem (the sunflower problem, the cake-cutting problem, etc.). Others point to some “killer application” where at least some subset of what the theorists produce is already being put to use directly in the world, like matching markets or auction design. But then there are the many theory papers that fall somewhere in the middle of the spectrum, where the theory is motivated by referring to some application or class of applications, without it being so clear that any such theory is already put to use for that application, and without the theorist attempting to produce results specific to that domain. From the outside looking in, it can be hard to judge how seriously readers are meant to take these connections, or how seriously the authors take them.
For example, in learning theory, applications are often mentioned to motivate new characterizations of bounds or optimal algorithms or solution concepts, typically at the beginning of a paper or talk, where examples like optimal treatment policies for healthcare or ad auctions or adaptive experimentation might be mentioned before the formal problem definition is given. Then maybe again at the end we hear about how slightly different classes of application motivate changing up some assumptions in future work. So the applications are like bookends, but not necessarily engaged with in any detail. Or sometimes, the intro and even related work sections of a paper seems to promise application-specificity (“This paper considers mechanism design for healthcare”) but then the application appears to get dropped once you get to the theory, with no looking back.
As a more applied person who can’t help but wonder about these loose ends, I’m often left feeling like I’m not fully appreciating the theory the way I’m supposed to. In watching theory talks, or reading theory papers (again usually on topics in the CS/econ/stats intersection), it’s not uncommon for me to reach a point about halfway through where I realize I no longer care much about the solution, because the pursuit of optimality or a complete characterization has taken over such that I can no longer relate the results to a real world problem. Or, maybe I still can with some effort, but then it seems impractical to think that anyone would want to try to apply the results in practice because using the theoretical framework adds so much complexity.
Related to this blog, I’m reminded of how a paper presenting a data hold-out mechanism for adaptive data analysis, patterned after differential privacy, once came up, but apparently didn’t work out so well when implemented. One of the authors of that work then confirmed that the work was a proof of concept, not really meant for practical application at the time of publication. Was it naive of the non-theorists to assume that a theoretical contribution motivated by a real world problem should be applicable to real examples of that problem at the time of publication? I don’t think so. It also seems fine for theorists to present theoretical solutions for practice problems even if they aren’t easy to apply in practice, as long as they are up front about that. I would hope that the average theorist, despite working in ‘theoryland’, wants more applied folks to take their contributions seriously and provide feedback. But the style of theory talks and papers often leads one to wonder if there is someone somewhere doing the follow-up work to see how well the thing can be applied in practice, what’s not trivial about it, what considerations might have been missed, etc.
In a panel on doing theory at the program I was at, someone mentioned how links between certain theory questions and real world applications can be taken for granted over time, even if the story is no longer very accurate. Would it be better not to mention the application at all if one isn’t sure of the applicability? Or is it unreasonable to expect authors working in a “practical” field like CS or econ to muster the level of confidence to stop mentioning the real world altogether?
I was recently reading Philip Stark’s deeply skeptical take on modelling for policy decisions, which is on some level the same spirit of questioning one poses at the more applied end of the spectrum. We take for granted in theory and modeling that certain sacrifices must be made in trying to make things work out given the tools we have. But what real world constraints we sever connections to can’t be taken lightly. So we need a lot of interchange between the domain experts working on the applications and the theorists, or someone whose focus is going between the two.
None of this is to say that theorists in the areas I mention aren’t aware or actively thinking about these questions. I heard several conversations at the Simon’s Institute that seemed to be about returning to or questioning the role of the motivating application to figure out how to proceed. My sense is that many theorists are quite aware when they are adding assumptions for tractability in finding a solution versus when assumptions or parts of a formulation are core to the problem itself and independent of the need to close the loop. My concern is more the ambiguity around how high priority the application is when looking in from the outside, e.g., reading theoretical papers that imply there is ultimately to be some bridging between theoryland and the real world but never get around to saying more. That seems like both the hardest and the most interesting part, but not necessarily well incentivized on either side.
I think it is mostly a matter of personal preference. I was trained as a theoretical economist – but I lost interest in that and mostly just wasn’t good enough to be successful. So, application interests me much more. The additional constraints that it imposes on a problem are mostly application specific – thus, of less interest to theorists, but that additional structure makes the problem much more interesting to me. Others may be much better and/or more interested in general solutions. In economics, it isn’t clear to me that there are many people capable of making much progress on general equilibrium theories anyway.
I guess the only thing that bothers me is when theorists tender conclusions to applied problems that are at odds with the relevant constraints of each particular problem. And, to be fair, applied analysts should not tender their analysis as a general analysis that applies when the particular constraints are not relevant. Truth in advertising should be practiced.
Agree, on the personal preference thing, and that its annoying when you get conclusions that seem opposed to basic constraint on the problem. It feels like the latter should be avoidable, like theorists should be responsible for developing (and presenting) an informed opinion about where they deviated from practice.
Jessica:
Your discussion is interesting, and it reminds me of the previous post on the “burly coolie” and the “titled English woman.” These were examples where the theory came first, and then a story was retrofitted to work with the theory.
After it was pointed out that the story was at worst completely made up and at best elaborated to a ridiculous, even parodical, degree, the authors were pushed to retract it, and they did so, adding, “the quote is in no way central to the core point of the paper, or even for the discussion in section VI of the paper. . . . Consequently, this incorrect quote can be omitted from the paper without any impact on the substance of the paper.”
The made-up (or, we can say, ridiculously elaborated) story fit the economic theory in the published paper, and it also fit a common meta-theory among economists that they speak out for the neglected common man. Kinda like how, in statistics, we tell stories that make our methods look good and that also support the self-image of statisticians as practical problem solvers.
Yes, it’s a good example.
And an unrelated story here. Well, not completely unrelated. Back when I worked at the University of California, my office was one or two doors down from an old guy, a professor who specialized in theoretical statistics. I have three stories about him:
1. I asked him once if there were any applications of the strong law of large numbers, which was one of his specialties. He replied, with equanimity, no.
2. When I was in the middle of working on Bayesian Data Analysis, I showed him a draft. I told him I was especially excited about the material in chapter 5 on hierarchical models. A couple days later I asked him what he thought. He replied that he had done this in the late 1940s in some application. I didn’t think to ask him why he hadn’t written it up, either then or later. Applied statistics is hard, but it’s not that hard, so I could believe that he independently discovered all this stuff but just thought it wasn’t interesting enough mathematically to write up. Or maybe he had done something completely different back then, I have no idea.
3. He would often walk outside the building to smoke. One day I asked if he’d ever thought of switching to chewing tobacco and putting a spittoon in his office. He said no, he hadn’t.
In theory, a spittoon should do it, but in practice, I guess not.
I’ve thought of this 3 story anecdote a few times over the past week, and it always cracks me up. There’s something I like about this guy (based on 3 stories). If it was just some guy with story 2 and 3, I wouldn’t have given it another thought. Story 1 sets everything up and makes his claim in story 2 believable and the question/response in 3 acceptable/expected.
Jd:
I found him kinda likable too, but I was also annoyed that he was so disrespectful of my work and then really mad when I found out he voted against promoting me. That said, I didn’t find him as annoying as the other colleague who stopped me in the hallway one day and told me that he heard that I was writing a book, and he didn’t think it was a good idea. Or the other colleague who asked me why, if my work on monitoring convergence from iterative simulations was such a good idea, I hadn’t written more papers on the topic. Or the colleague who came up to me after a talk I gave on modeling elections and asked me why, given that votes are discrete, I wasn’t using a generalized linear model. Or the colleague who came up to me after a different voting-related talk and confided to me that he’d never realized how trivial that political science research was. Or the colleagues who wrote a report lying about my work, among other things saying that all I’d done was fit linear models, even though I’d sent them a paper that used a nonlinear differential equation model. Or the external letter writer who wrote that I had multiple mistakes in my book, and then when I asked to be told exactly what these mistakes were, failed to respond.
Compared to those people, a courtly guy who just didn’t see any value to my work, that wasn’t so bad at all. He really was charming. He just had no use for applied statistics.
Sounds like a frustrating place to work.
“He just had no use for applied statistics.”
Huh. Seems like a difficult view to hold. Having no use for applied mathematics sounds easier. Whenever I think of statistics, I think of application by definition (deals with data).
“But the style of theory talks and papers often leads one to wonder if there is someone somewhere doing the follow-up work to see how well the thing can be applied in practice, what’s not trivial about it, what considerations might have been missed, etc.”
I think this is a type of researcher that exists–applied, familiar with theory but not a theorist, and focused on applying theoretically state of the art methods to real problems and working out the kinks. They pave the way for other applied researchers to use these methods by conducting illustrative early adoption applications. This is common in causal inference and I think it’s super important.
“In watching theory talks, or reading theory papers (again usually on topics in the CS/econ/stats intersection), it’s not uncommon for me to reach a point about halfway through where I realize I no longer care much about the solution, because the pursuit of optimality or a complete characterization has taken over such that I can no longer relate the results to a real world problem.”
This articulates very well the conclusion I drew when I was studying microeconomic theory. I ended up moving into industrial organization, a subset of “applied micro” that still has plenty of theory work, because there is a much stronger culture of maintaining a connection to the motivating questions and especially to the specific industry being studied.
I’ve also found that macroeconomic theory tends to maintain a fairly strong connection to real world problems, although I had too many objections to the (very rigid) modeling paradigms to remain with it.
I’m with you. Micro theory has plenty of unsolved problems, but true theoretical developments are beyond my reach. Application via industrial organization, on the other hand, really only makes sense when applied to real markets. As for macro, I see the problem differently – plenty of applied problems and no theories (at least none that make any sense to me).
Thanks Jessica for the interesting post. I will add a data point.
I grew up deep in theoryland (pure math) and even there it’s very common for papers to tout their “applications”. Someone will write a paper on expanders (a type of graph) and the intro will claim that this is useful for designing fault tolerant networks. Well, yeah, but if you just draw a network at random there’s a 99.9% chance it’ll be fault tolerant (this is a theorem btw), so, you don’t really need any more theory about it for practical purposes. Same goes for pretty much every sub area of pure math. (Logicians might be the only exception?)
I point this out to show that there’s a large body of people out there making applicability claims, who don’t really mean them, and absolutely cannot substantiate them if pushed. So, I wouldn’t be surprised if other people are also doing it.
I’ve always read theoretical CS papers in the same spirit, regarding the applicability remarks as a kind of funny ritual our tribe does to feel good about ourselves.
Theory research is about expanding the logical bounds of known generalizations into the unknown in an internally consistent way. On the far theoretical end of the spectrum, it’s main application is testing the soundness of existing theory. Can’t speak for CS but my experience elsewhere is that people work across the spectrum from pure theory to pure application and from pure theory to pure empiricism. I guess if there’s no one working in the center of the triangle in CS it’s an opportunity?
I get your thing about papers that promise something and then completely ignore it! That made me laugh. A similar thing is papers that present great evidence for [alternative hypothesis], then at the end of the discussion suddenly say out of the blue “we find our research offers strong support for [longstanding dogma]”. It’s like Huh?
I loved the first line in Phillip Stark’s abstract:
“Many widely used models amount to an elaborate means of making up numbers”
badabing.
Every once in a while theory & practice match well.
In 1962, in high school, I attended a National Youth Conference on the Atom, of which one part was lectures by ~dozen scientists.
One was Henry Pollak, Director of Bell Labs Mathematics & Statistics Research Center, which included folks like Joe Kruskal and John Chambers.
That was in a Division run by Executive Director Robert Prim, Associate Exec Dir John Tukey.
Henry gave a very lucid talk that quickly attracted at least half of the students.
He said: here’s a nice graph theory problem, minimal spanning tree (algorithms by Prim and Kruskal), then posed a question: what’s it good for?
Answer, of course it was used heavily in planning telecommunications networks so save much money for the Bell System.
https://en.wikipedia.org/wiki/Minimum_spanning_tree
https://en.wikipedia.org/wiki/Henry_O._Pollak
Anyway, I never forgot the idea that sometimes theory really helps practice.
Another tendency is to start with an existing solution to a practical problem, then add some theory and “generalize it” with the previous solution as a special case. Yet, when you try and come up with specific applications, you struggle t imagine any examples other than the case that had already been solved. For example, I think the field of topological data analysis spun up and wound down rapidly with lots of effort and not much in the way of practical success. It turns out a lot of the interesting content in real data sets is not metric-invariant, so forgetting a metric kind of means ignoring the content! It kind of reminds me of the string theory debacles and Wolfram’s New Kind of Science (TM). It’s a theory that can explain things that we do, and also could explain things that we don’t do. Generously, I guess it can help a certain kind of brain wrap around a problem. Ungenerously, it’s a paper generating algorithm.