This is Jessica. Recall that in 2020, NeurIPS added a requirement that authors include a statement of ethical aspects and future societal consequences extending to both positive and negative outcomes. Since then, requiring broader impact statements in machine learning papers has become a thing.
The 2024 NeurIPS call has not yet been released, but in 2023 authors were required to complete a checklist where they had to respond to the following: “If appropriate for the scope and focus of your paper, did you discuss potential negative societal impacts of your work?”, with either Y, N, or N/A with explanation as appropriate. More recently, ICML introduced a requirement that authors include impact statements in submitted papers: “a statement of the potential broader impact of their work, including its ethical aspects and future societal consequences. This statement should be in a separate section at the end of the paper (co-located with Acknowledgements, before References), and does not count toward the paper page limit.”
ICML provided authors who didn’t feel they had much to say the following boiler-plate text:
“This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.”
but warned authors to “to think about whether there is content which does warrant further discussion, as this statement will be apparent if the paper is later flagged for ethics review.”
I find this slightly amusing in that it sounds like what I would expect authors to be thinking even without an impact statement: This work is like, so impactful, for society at large. It’s just like, really important, on so many levels. We’re out of space unfortunately, so we’ll have to leave it at that.\newline\newline\newline\newline Love, \newline\newline\newline\newline the authors \newline\newline\newline\newline
I have an idea that might increase the value of the exercises, both for authors and those advocating for the requirements: Have authors address potential impacts in the context of their discussion of related work *with references to relevant critical work*, rather than expecting them to write something based on their own knowledge and impressions (which is likely to be hard for many authors for reasons I discuss below). In other words, treat the impact statement as another dimension of contextualizing one’s work against existing scholarship, rather than a free-form brainstorm.
Why do I think this could be an improvement? Here’s what I see as the main challenges these measures run into (both my own thoughts and those discussed by others):
- Lack of incentives for researchers to be forthright about possible negative implications of their work, and consequently a lack of depth in the statements they write. Having them instead find and cite existing critical work on ethical or societal impacts doesn’t completely reconcile this, but presumably the critical papers aren’t facing quite the same incentives to say only the minimum amount. I expect it is easier for the authors to refer to the kind of critiques that ethics experts think are helpful than it is for them to write such critical reflections themselves.
- Lack of transparency around how impacts statements factor into reviews of papers. Authors perceive reviewing around impacts statements as a black box, and have responded negatively to the idea that their paper could potentially get rejected for not sufficiently addressing broader impacts. But authors have existing expectations about the consequences for not citing some relevant piece of prior work.
- Doubts about whether AI/ML researchers are qualified to be reflecting on the broader impacts of their work. Relative to say, the humanities, or even areas of computer science that are closer to social science, like HCI, it seems pretty reasonable to assume that researchers submitting machine learning papers are less likely to gravitate to and be skilled at thinking about social and ethical problems, but skilled at thinking about technical problems. Social impacts of technology require different sensibilities and training to make progress on (though I think there are also technical components to these problems as well, which is why both sides are needed). Why not acknowledge this by encouraging the authors to first consult what has been said by experts in these areas, and add their two cents only if there are aspects of the possible impacts or steps to be taken to address them (e.g., algorithmic solutions) that they perceive to be unaddressed by existing scholarship? This would better acknowledge that just any old attempt to address ethics is not enough (consider, e.g., Gemini’s attempt not to stereotype, which was not an appropriate way to integrate ethical concerns into the tech). It would also potentially encourage more exchange between what currently can appear to be two very divided camps of researchers.
- Lack of established processes for reflecting on ethical implications in time to do something about them (e.g., choose a different research direction) in tech research. Related work is often one of the first sections to be written in my experience, so at least those authors who start working on their paper in advance of the deadline might have a better chance of acknowledging potential problems and adjusting their work in response. I’m less convinced that this will make much of a difference in many cases, but thinking about ethical implications early is part of the end goal of requiring broader impacts statements as far as I can tell, and my proposal seems more likely to help than hurt for that goal.
The above challenges are not purely coming from my imagination. I was involved in a couple survey papers led by Priyanka Nanayakkara on what authors said in NeurIPS broader impacts statements, and many contained fairly vacuous statements that might call out buzzwords like privacy or fairness but didn’t really engage with existing research. If we think it’s important to properly understand and address potential negative societal impacts of technology, which is the premise of requiring impacts statements to begin with, why expect a few sentences that authors may well be adding at the last minute to do this justice? (For further evidence that that is what’s happening in some cases, see e.g., this paper reporting on the experiences of authors writing statements). Presumably the target audience of the impact statements would benefit from actual scholarship on the societal implications over rushed and unsourced throwing around of ethical-sounding terms. And the authors would benefit from having to consult what those who are investing the time to think through potential negative consequences carefully have to say.
Some other positive byproducts of this might be that the published record does a better job of pointing awareness to where critical scholarship needs to be further developed (again, leading to more of a dialogue between the authors and the critics). This seems critical, as some of the societal implications of new ML contributions will require both ethicists and technologists to address. And those investing the time to think carefully about potential implications should see more engagement with their work among those building the tools.
I described this to Priyanka, who also read a draft of this post, and she pointed out that an implicit premise of the broader impact requirements is that the authors are uniquely positioned to comment on the potential harms of their work pre-deployment. I don’t think this is totally off base (since obviously the authors understand the work at a more detailed level than most critics), but to me it misses a big part of the problem: that of misaligned incentives and training (#1, #3 above). It seems contradictory to imply that these potential consequences are not obvious and require careful reflection AND that people who have not considered them before will be capable of doing a good job at articulating them.
At the end of the day, the above proposal is an attempt to turn an activity that I suspect currently feels “religious” for many authors into something they can apply their existing “secular” skills to.
Jessica:
The last sentence of your post reminds me of something I’ve thought a lot at with statistical research, which is the development of ideas that bring things from the zone of “philosophy” or “good practice” toward “method.”
Some examples:
– Posterior predictive checking. Instead of saying that the model is subjective, full stop, we develop methods for checking the fit of model to data.
– Graphs as comparisons and exploratory data analysis as comparison to an implicit model. Instead of EDA being something that should be done in some vague way, it becomes more closely tied to inference, which is traditionally taken to be the more serious part of statistics.
– Multilevel modeling. Instead of this weird thing called “empirical Bayes,” which gives the awkward and incorrect impression that full Bayes is non-empirical, we put estimation of hyperparameters inside the full Bayesian inference process.
– Including group-level predictors in multilevel models. Instead of relying on “exchangeability” as a philosophical or quasi-mystical idea, which leads to empty statements such as, “I don’t believe in exchangeability here,” we frame violations of exchangeability in terms of the presence of information which can then be included in the model. Again, what was formerly a province of philosophy is now subsumed into statistical method.
– Rubin’s general framework of missing data. Instead of missingness being some annoying thing that needs to be handled on a one-off basis, we frame problems of prediction, sampling, causal inference, rounding, and measurement error into a unified “missing data” framework.
Each of these developments has two advantages: first, a more general theoretical framework allows us to better understand what we are doing and to develop new methods; second, it takes steps that were formerly outside the formal workflow and puts it in.
Great examples. I have always gravitated toward work that takes something that seems nebulous, or that people perceive as being essentially unformalizable (e.g., “storytelling”, “EDA”) and makes it more concrete. I’ve tried to do that with some of my own research too.
In this case, reflecting on the future negative societal implications of new technology seems inherently nebulous, but describing what has already been said about negative consequences of related technology (and based on this, reflecting on what might be unique to one’s own work) seems like a starting point that is attainable.
Jessica:
Here’s a “stastistical” method of assessing the “social consequence” of some new technological development: take a data table with one million lines of data for one million variables. Select 20 variables that are currently in the popular imagination of a small segement of society. Eyeball the outcome of the effect of the technology on some of these 20 variables. It makes random variable comparisons and reporting of “statistical significance” look like genius science.
A more sensible way to think about “social consequences” of technology is this: people don’t invent things that have strongly negative consequences. They invent things to make life better – that’s why humans haven’t wiped themselves out yet. It’s a protection already coded into the system. Even weapons are in this category, because they are used to protect people from other people. So the net effect is that *virtually every invention* has a *far* greater chance of having a positive social outcome than a negative one. IOW, there is an extremely strong bias for beneficial technological developments.
Imagine this situation: suppose a development comes along that creates a large increase in GDP but has a negative effect on employment of some segment of the population (SOP). Is that a “negative” or a “positive” for that SOP? How do you know the net effect? Suppose the excess returns of GPD are invested and ultimately produce major advances and cost reductions in health care, so that, despite having less employment, the relevant SOP has better health care? Pete B talks about “racist freeways” but those “racist freeways” carry lots of poor people to hospitals in ambulances where their care is paid by the government.
unhelpful
By your measure, no impact study is ever necessary and every new technology is – by definition – an improvement. I would emphasize the “yet” in your diatribe. Nuclear weapons certainly have some positive potential. I don’t sleep better at night because of that and the “yet” does disturb me. Let’s not forgot the massive loss of life that has already occurred due to that technology (arguably, to prevent a larger loss of life, but certainly debatable).
You are certainly correct that virtually any new technology has the potential for both good and bad impacts. I do not accept the assumption that the net benefits are positive – there are plenty of bad actors whose motivations involve a private calculus that differs from the social calculus. More importantly, you are advocating not burdening developers of technology with any need to think about their actions and the consequences of their actions.
There is an element of these impact study requirements that bothers me. Usually the bad actors are not affected by them, other than the cost of compliance. That cost also affects all those who would not have pursued dangerous research to begin with. Much relies on the belief that these requirements will cause people to think more carefully about the potential impacts of their research than they normally would, and on the belief that such thinking is needed because it is so hard to predict the ways new technology might be used. Both of these assumptions are at play with AI – and I’m not sure they are evidence based. But your evidence based conclusion that there is no need because “humans haven’t wiped themselves out yet” is extraordinary weak. Many humans have been wiped out. The criterion should not be whether we have become extinct or not. Tragedies can occur without extinction (and for many species, extinction has actually occurred).
chipmunk
> So the net effect is that *virtually every invention* has a *far* greater chance of having a positive social outcome than a negative one.
As long as you’re sufficiently selective that’s true.
But it’s intersting that you’re on the lab-mediated spillover side of the COVID origins debate. I would have guessed otherwise.
Ethical considerations are fine. Institutionalized ethical considerations is code for mandating, banning, censoring and whatever else maintains the status quo power structure. Just another layer of counterproductive peer review.
There is definitely an element of power struggle as far as I can tell. I’m not sure the goal is censorship, but the idea that this is a manifestation of some sort of power structure seems reasonable.
I can’t really interpret it beyond the more obvious intention to get people thinking about and explicitly recording possible implications. There’s sort of a leap of faith that seems required for someone to really get incentivized to write a good statement, which is why I think it makes more sense to set it up in a way that authors can apply skills and expectations they already have about related work.
Could be some well-meaning attempt at something good, but more gatekeeping is the inevitable result.
Abd afaict society already had this discussion with the rise of facebook, google, twitter, etc. The public apparently decided they don’t care about being spied on and manipulated by “psychopathic” optimizing algorithms.
What discussion can be had about AI that doesn’t already apply to the behavior of corporations?
I agree there’s truth to this, but at the same time we can’t let that scare us from openly thinking about and discussing ethics. Not that I think you were implying the opposite.
I still feel as though authors may require examples a d structured prompts to consider broader societal impacts in Related Work sections. Is there an opportunity for an empirical study here with graduate students in CS/HCI?
Or else, the task could be left to reviewers.
It seems like this provides pretty structured guidance: https://medium.com/@GovAI/a-guide-to-writing-the-neurips-impact-statement-4293b723f832
And most of the “good” examples do include citations, so I’m not sure what the strong reason is not to make this part of related work
In terms of a study, my hypothesis that the process of writing a statement is easier for authors and leads to better output when approached as a part of related work versus as a stand-alone section could be evaluated empirically.
What evidence is there, after all these thousands of hours have been spent by authors/reviewers/readers, that any of these impact statements have done anything but pad papers with boilerplate? Shouldn’t we strongly assume, given how little ‘impact’ there is from social science experiments (as documented for decades on this blog), even ones vastly more invasive than impact statements, that the effect is null?
Instead of making suggestions in the grounds that they sound nice and it seems intuitive that they would help and you want them to help, which is unscientific, perhaps it would be better to discuss how to go about getting evidence that they work? For example, randomizing half the NIPS papers to omitting it (I’m sure most of the authors won’t mind), and seeing if the ‘impacts’ a year later seem any less ethical or less societally-affecting.
I think you are mistaking me for being an advocate for these statements. All I am saying is that if we are going to require them, there would seem to be a better way to do it. I don’t really see my personal opinion about their value as being likely to change anyone’s mind (I’m not that involved with machine learning research). But maybe someone will see this suggestion and the experience will become at least slightly less painful for authors.
You seem to be advocating for them given that your conclusion, after involvement in work that concludes that they don’t work, that since ‘violence isn’t solving the problem we just need to use more of it’, and calling for more labor put into impact statements.
And I don’t think your suggestion is good, particularly given your results about authors’ response to the imposition of impact statements.
It is a recipe for citation manufacturing and logrolling at scale, by requiring boilerplate which is no longer harmless empty verbiage but contains citations, which will be copy-pasted ad nauseam no matter the quality, because now it’s a requirement.
Imagine every single paper at NIPS frantically reaching at the deadline for some paper, any paper, which “references to relevant critical work”, even when said critical work doesn’t exist (eg. because you’re doing novel research) or where the work in question is bad.
This would also incentivize a gold rush to plant flags and become ‘the’ CNN “relevant critical work” paper, or ‘the’ Transformer critical work paper, so as to get into the first batch of papers that authors & readers will copy-paste from thereon out. (We already see this in DL. There are certain papers which get cited to provide the expected-but-not-required ‘criticism’ citation, though they were obviously wrong even when published, and have not aged any better than that sounds; and it’s bad for the good researchers doing this critical work too, because there are generally newer critical papers which are much better but get ignored in favor of the ‘canonical’ ones from 2020 or so.)
At least when authors write “fairly vacuous statements”, it merely wastes time. But once you supercharge the perverse incentives, I think you’ll find that it is true that “presumably the critical papers aren’t facing quite the same incentives to say only the minimum amount”, but you won’t like the amounts that their incentives produce either…
The work I was involved in didn’t exactly conclude they didn’t work. There are a lot of people who think the evidence of statements so far is positive. They just don’t seem to like the resistance. I doubt it’s a useful exercise, but I also wouldn’t go so far as to liken the requirement to violence.
I can see how my post might be interpreted as supporting them, but I have accepted that they are probably here to stay. While I don’t see much value in them currently, I am open to the possibility that given that some technology does motivate serious thought about social implications (the most obvious example being self-driving cars), encouraging more dialogue between ML researchers and philosophers/ethicists/etc could be useful. I don’t think the critics/ethicists are going to solve the problems themselves without getting into technical issues, and so some back and forth seems important for the truly risky cases. Maybe that’s never going to happen through a requirement put on all authors, though.
For the contributions where there is minimal risk, regardless of how you frame the requirement, I assume those authors still have the option of not addressing impacts. So while I can see how having to add citations could create its own industry and more work for authors, I’m less pessimistic that putting the section in related work would change the entire ballgame.