Since Aki and Andrew are doing it…
Published:
- Dongping Zhang, Jason Hartline, and Jessica Hullman (2024). Designing Shared Information Displays for Agents of Varying Strategic Sophistication. ACM Transactions on Computer-Supported Cooperative Work (CSCW).
- Yifan Wu, Ziyang Guo, Michalis Mamakos, Jason Hartline, and Jessica Hullman (2023). The rational agent benchmark for data visualization. IEEE Transactions on Visualization & Computer Graphics (Proc. VIS ‘23).
- Alex Kale, Ziyang Guo, Xiao Li Qiao, Jeff Heer, Jessica Hullman (2023). EVM: Incorporating model checking into exploratory visual analysis. IEEE Transactions on Visualization & Computer Graphics (Proc. VIS ‘23).
- Hari Subramonyam and Jessica Hullman (2023). Are We Closing the Loop Yet? Gaps in the Generalizability of VIS4ML Research. IEEE Transactions on Visualization & Computer Graphics (Proc. VIS ‘23).
- Hyeok Kim, Ryan Rossi, Jessica Hullman, and Jane Hoffswell (2023). Dupo: A Mixed-Initiative Authoring Tool for Responsive Visualization. IEEE Transactions on Visualization & Computer Graphics (Proc. VIS ‘23).
- Fumeng Yang, Mandi Cai, Chloe Mortenson, Hoda Fakhari, Ayse Deniz Lokmanoglu, Jessica Hullman, Steven Franconeri, Nicholas Diakopoulos, Erik Nisbet, and Matthew Kay. Swaying the Public? Impacts of Election Forecast Visualizations on Emotion, Trust, and Intention in the 2022 US Midterms. IEEE Transactions on Visualization & Computer Graphics (Proc. VIS ‘23).
- Andrew Gelman, Jessica Hullman, and Lauren Kennedy (2023). Causal quartets: Different ways to attain the same average treatment effect. American Statistician.
- Priyanka Nanayakkara and Jessica Hullman (2023). What’s driving conflicts around differential privacy for the US Census. IEEE Security & Privacy.
- Alex Kale, Sarah Lee, T.J. Goan, Beth Tipton, and Jessica Hullman (2023). MetaExplorer: Facilitating Reasoning with Epistemic Uncertainty in Meta-analysis. ACM Transactions on Computer-Human Interaction (Proc. CHI ‘23).
- Abhraneel Sarma, Alex Kale, Michael Jongho Moon, Nathan Taback, Fanny Chevalier, Jessica Hullman, and Matthew Kay (2023). multiverse: Multiplexing alternative data analyses in R notebooks. ACM Transactions on Computer-Human Interaction (Proc. CHI ‘23).
Unpublished/Preprints:
- Jake Hofman, Angelos Chatzimparmpas, Amit Sharma, Duncan Watts, and Jessica Hullman. Pre-registration for Predictive Modeling.
- Jessica Hullman, Ari Holtzman, and Andrew Gelman. Artificial Intelligence and Aesthetic Judgment.
- Sayash Kapoor, Emily Cantrell, Kenny Peng, Thanh Hien Pham, Christopher A. Bail Odd Erik Gundersen, Jake M. Hofman, Jessica Hullman, Michael A. Lones, Momin M. Malik, Priyanka Nanayakkara, Russell A. Poldrack, Inioluwa Deborah Raji, Michael Roberts, Matthew J. Salganik, Marta Serra-Garcia, Brandon M. Stewart, Gilles Vandewiele, and Arvind Narayanan. Reforms: Reporting standards for machine learning based science.
- Jessica Hullman. Some problems with zooming out as science reform (commentary on Almaatouq et al.).
Performed:
- Andrew Gelman and Jessica Hullman. Recursion (a play). Performed by a cast of Northwestern University Theater students at ACM Conference on Fairness, Accountability, and Transparency in Artificial Intelligence (FAccT ’23). Chicago, IL.
If I had to choose a favorite (beyond the play, of course) it would be the rational agent benchmark paper, discussed here. But I also really like the causal quartets paper. The first aims to increase what we learn from experiments in empirical visualization and HCI through comparison to decision-theoretic benchmarks. The second aims to get people to think twice about what they’ve learned from an average treatment effect. Both have influenced what I’ve worked on since.
Nice! I also really liked the causal quartet paper.
I don’t work in visualization research (at least formally, but I do make lots of plots for human consumption!), but reading the rational agent benchmark paper really makes me realize how difficult it is to research these topics! Generally, I make a plot, and then I either 1) ask several people to tell me what they learned, or 2) send it to them and wait until the discussion meeting to see what questions they have (pretty revealing sometimes, especially if I did a poor job translating info to consumable visual). It’s a lot easier to make plots for my own consumption vs someone else’s…
I think in many applied settings your best bet is to try to understand how the people you designed the vis for actually use it – you can get a lot of valuable information if your users are invested in helping you improve it for their purposes. But visualization interpretation has also become a research topic in its own right, and so there are more and more people running controlled experiments to try to establish what aspects of a visualization design or scenario lead to what kind of performance. The belief is that this work is more rigorous and allows for the development of generalizable theories about what makes an effective visualization. But it is often conducted without there being a well-defined standard for “good” performance (whether inference or decision-making) to compare people’s performance to. So the interpretations of the results can be misleading. It’s not clear to me looking at it that we’re getting closer to understanding what makes a visualization effective for reasoning under uncertainty. And all this is also playing out (with even more hype) in the growing base of research on how to design for AI-assisted decision making. My next benchmark paper will tackle that!
The Reforms paper was nice to see as someone interested in reporting guidelines. Though by now, it seems like there are so many forming that a wiki would be a good idea to really scale up the practice. It would also make it easier to collect suggestions from diverse readers and evolve guidelines more readily.
I also know a toxicologist who’s infusing reporting guidelines in the paper authoring stage to ensure that scholars actually use these guidelines.
We need a checklist to keep track of all the checklists.
I guess you find it pretty annoying when someone’s simple idea is a lot more powerful than your complicated one.
Apologies if I misinterpreted your comment. If you are making a suggestion about the quartets (you mention they “need tweaks”) perhaps you could try restating what the suggestion is?
Thanks for clarifying. And for the recs on who to look out for at Northwestern!