2024 was an enjoyable year. Below are a few things I did.

Conference and journal papers published

• Improving out-of-population prediction: The complementary effects of model assistance and judgmental bootstrapping. International Journal of Forecasting. Hardy, M. D., Zhang, S., Hullman, J., Hofman, J. M., and Goldstein, D. G. 
• What to Consider When Considering Differential Privacy for Policy. Policy Insights from the Behavioral and Brain Sciences (PIBBS). Nanayakkara, P. and Hullman, J.
• VMC: A Grammar for Visualizing Statistical Model Checks. IEEE Transactions of Visualization & Computer Graphics (Proceedings of IEEE VIS 2024). Guo, Z., Kale, A., Kay, M., and Hullman, J.
• REFORMS: Consensus-based Recommendations for Machine-learning-based Science. Science Advances, 10 (18). Kapoor, S., Cantrell, E. M., Peng, K., Pham, T. H., Bail, C., Gundersen, O. E., Hofman, J., Hullman, J., Lones, M., Malik, M., Nanayakkara, P., Poldrack, R., Raji, I. D., Roberts, M., Salganik, M., Serra-Garcia, M., Stewart, B., Vandewiele, G., and Narayanan, A.
• A Conceptual Framework for Ethical Evaluation of Machine Learning Systems. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. Gupta, N. R., Hullman, J., and Subramonyam, H.
• Measure-Observe-Remeasure: An Interactive Paradigm for Differentially-Private Exploratory Analysis. IEEE symposium on Privacy & Security (Proceedings of S&P 2024). Nanayakkara, P., Kim, H., Wu, Y., Sarvghad, A., Mahyar, N., Miklau, G., and Hullman, J. 
• A Decision Theoretic Framework for Measuring AI Reliance. ACM Conference on Fairness, Accountability, & Transparency (Proceedings of FAccT 2024). Guo, Z., Wu, Y., Hartline, J., and Hullman, J.
• Evaluating the Utility of Conformal Prediction Sets for AI-Advised Image Labeling. ACM Conference on Human Factors in Computing Systems (Proceedings of CHI 2024). Zhang, D., Chatzimparmpas, A., Kamali, N., and Hullman, J.
• Erie: A Declarative Grammar for Data Sonification. ACM Conference on Human Factors in Computing Systems (Proceedings of CHI 2024. Kim, H., Kim, Y-S., and Hullman, J.
• Milliways: Taming Multiverses through Principled Evaluation of Data Analysis Paths. ACM Conference on Human Factors in Computing Systems (Proceedings of CHI 2024. Sarma, A., Hwang, K., Hullman, J., and Kay, M.
• Causal quartets: Different ways to attain the same average treatment effect. American Statistician 78. Gelman, A., Hullman, J., and Kennedy, L.

Three of my collaborators above were Ph.D. students I advised, who graduated in 2024! Congrats to Priyanka Nanayakkara (now a postdoc at Harvard CRCS), Hyeok Kim (now a postdoc at University of Washington CS, on the academic job market), and Dongping Zhang (now a research scientist at NREL). 

Talks (that are available online)

• Benchmarking Visualization for Decision-Making. MIT Code Conference, Oct. 2024.
• Data analysis and imagination. Alan Turing Institute, June 2024.

I gave a several other talks that I think exist somewhere online, but I can’t find the links. 

Workshop participation/organization 

The highlights of the year for me were workshops I attended over the summer. Getting to travel to interesting places to think deeply about topics you find fascinating is truly a privilege. 

• Bridging Prediction and Intervention Problems in Social Systems. Organized by Lydia Liu, Inioluwa Deborah Raji, Angela Zhou, and Arvind Narayanan. Banff Canada, June 2024. 
• Navigating the garden of forking paths/Theoretical foundations for interactive data analysis in data-driven science. Organized by Cagatay Turkay and Roger Beecham. London, June 2024.
• Workshop on Individualized Decision-Making. Organized by Ben Recht. Berkeley, CA, July 2024.
• Apple Workshop on Human-Centered Machine Learning. Organized by Kareem Bedri, Leah Findlater, Dominik Moritz, and Jeff Nichols. Cupertino, CA, Aug 2024.

I co-organized a few other workshops I enjoyed:

• Theoretical Foundations of Human-AI Complementarity. With Jason Hartline. Northwestern University, Sept. 2024.
• Statistical Frontiers in LLMs and Foundation Models. With Anastasios Angelopoulos, Stephen Bates, Alex D’Amour, Fanny Yang, Sophia Sun, and Tatsunori Hashimoto. NeurIPS, Vancouver, Dec. 2024.

My goal for next year is to do more creative writing. I would be thrilled if I could write even a couple poems I’m happy with.