Hand-drawn Statistical Workflow at Nelson Mandela

In September 2023 I taught a week-long course on statistical workflow at the Nelson Mandela African Institution of Science and Technology (NM-AIST), a public postgraduate research university in Arusha, Tanzania established in 2009.

NM-AIST – CENIT@EA

The course was hosted by Dean Professor Ernest Rashid Mbega and the Africa Centre for Research, Agricultural Advancement, Teaching Excellence and Sustainability (CREATES) through the Leader Professor Hulda Swai and Manager Rose Mosha.

Our case study was an experiment on the NM-AIST campus designed and implemented by Dr Arjun Potter and Charles Luchagula to study the effects of drought, fire, and herbivory on growth of various acacia tree species. The focus was pre-data workflow steps, i.e. experimental design. The goal for the week was to learn some shared statistical language so that scientists can work with statisticians on their research.

Together with Arjun and Charles, with input from Drs Emmanuel Mpolya, Anna Treydte, Andrew Gelman, Michael Betancourt, Avi Feller, Daphna Harel, and Joe Blitzstein, I created course materials full of activities. We asked participants to hand-draw the experimental design and their priors, working together with their teammates. We also did some pencil-and-paper math and some coding in R.

Course participants were students and staff from across NM-AIST. Over the five days, between 15 and 25 participants attended on a given day.

Using the participants’ ecological expertise, we built a model to tell a mathematical story of how acacia tree height could vary by drought, fire, herbivory, species, and plot location. We simulated parameters and data from this model, e.g. beta_fire = rnorm(n = 1, mean = -2, sd = 1) then simulated_data …= rnorm(n, beta_0 + beta_fire*Fire +… beta_block[Block], sd_tree). We then fit the model to the simulated data.

Due to difficulty in manipulating fire, fire was assigned at the block-level, whereas drought and herbivory were assigned at the sub-block level. We saw how this reduced precision in estimating the effect of fire:

We redid the simulation assuming a smaller block effect and saw improved precision. This confirmed the researcher’s intuitions that they need to work hard to reduce the block-to-block differences.

To keep the focus on concepts not code, we only simulated once from the model. A full design analysis would include many simulations from the model. In Section 16.6 of ROS they fix one value for the parameters and simulate multiple datasets. In Gelman and Carlin (2014) they consider a range of plausible parameters using prior information. Betancourt’s workflow simulates parameters from the prior.

Our course evaluation survey was completed by 14 participants. When asked “which parts of the class were most helpful to you to understand the concepts?”, respondents chose instructor explanations, drawings, and activities as more helpful than the R code. However, participants also expressed eagerness to learn R and to analyze the real data in our next course.

The hand-drawn course materials and activities were inspired by Brendan Leonard’s illustrations in Bears Don’t Care About Your Problems and I Hate Running and You Can Too. Brendan wrote me,

I kind of think hand-drawing stuff makes it more fun and also maybe less intimidating?

I agree.

More recently, I have been reading Introduction to Modern Causal Inference by Alejandro Schuler and Mark van der Laan, who say

It’s easy to feel like you don’t belong or aren’t good enough to participate…

yup.

To deal with that problem, the voice we use throughout this book is informal and decidedly nonacademic…Figures are hand-drawn and cartoonish.

I’m excited to return to NM-AIST to continue the workflow steps with the data that Dr Arjun Potter and Charles Luchagula have been collecting. With the real data, we can ask: is our model realistic enough to achieve our scientific goals ?

Blue Rose Research is hiring (yet again) !

Blue Rose Research has a few roles that we’re actively hiring for as we gear up to elect more Democrats in 2024, and advance progressive causes!

A bit about our work:

  • For the 2022 US election, we used engineering and statistics to advise major progressive organizations on directing hundreds of millions of dollars to the right ads and states.
  • We tested thousands of ads and talking points in the 2022 election cycle and partnered with orgs across the space to ensure that the most effective messages were deployed from the state legislative level all the way up to Senate and Gubernatorial races and spanning the issue advocacy space as well.
  • We were more accurate than public polling in identifying which races were close across the Senate, House, and Gubernatorial maps.
  • And we’ve built up a technical stack that enables us to continue to build on innovative machine learning, statistical, and engineering solutions.

Now as we are looking ahead to 2024, we are hiring for the following positions:

All positions are remote, with optional office time with the team in New York City.

Please don’t hesitate to reach out with any questions ([email protected]).

Blue Rose Research is hiring (again) !

Blue Rose Research has a few roles that we’re actively hiring for as we gear up to elect more Democrats in 2024, and advance progressive causes!

A bit about our work:

  • For the 2022 US election, we used engineering and statistics to advise major progressive organizations on directing hundreds of millions of dollars to the right ads and states.
  • We tested thousands of ads and talking points in the 2022 election cycle and partnered with orgs across the space to ensure that the most effective messages were deployed from the state legislative level all the way up to Senate and Gubernatorial races and spanning the issue advocacy space as well.
  • We were more accurate than public polling in identifying which races were close across the Senate, House, and Gubernatorial maps.
  • And we’ve built up a technical stack that enables us to continue to build on innovative machine learning, statistical, and engineering solutions.

Now as we are looking ahead to 2024, we are hiring for the following positions:

All positions are remote, with optional office time with the team in New York City.

Please don’t hesitate to reach out with any questions ([email protected]).

Blue Rose Research is hiring !

Blue Rose Research is looking to expand as we gear up to elect more Democrats in 2024, and advance progressive causes !

A bit about our work:

  • In the 2022 US elections, we used engineering and statistics to advise major progressive organizations on directing hundreds of millions of dollars to the right ads and states.

  • We tested thousands of ads and talking points in the 2022 election cycle and partnered with over a hundred organizations across the space to ensure that the most effective messages were deployed in hundreds of house, senate, gubernatorial, and state legislative races across the country – in addition we played a large role in helping issue advocacy groups and ballot measure campaigns ranging from reproductive rights to medicaid expansion.

  • We were more accurate than public aggregators in identifying which races were close across the Senate, House, and Gubernatorial maps.

  • And we’ve built up a technical stack that enables us to continue to build on innovative machine learning, statistical, and engineering solutions.

Now as we are looking ahead to 2024, we are hiring for the following positions:

All positions are remote, with optional office time with the team in New York City.

Please don’t hesitate to reach out with any questions ([email protected]).

preregistered vs adaptable Bayesian workflow, and who should do the work ?

Johannes Haushofer asks:

Later in the thread, the Millennium Villages Project (MVP) evaluations came up:

I thought:

1. Our MVP evaluation says:

The design was completed before endline data collection, and a peer-reviewed evaluation protocol was registered with The Lancet.

Could the protocol have been more detailed and constrained more researcher degrees of freedom ? Yes.

This may not be a solution though, as Michael Betancourt writes:

Preserving a preregistered model only perpetuates its optimistic assumptions about the realization of an experiment. In practice we almost always have to adapt our model to the peculiarities of the observed data.

The Bayesian workflow authors say:

Our claim is that often a model whose assumptions withstood such severe tests is, despite being the result of data-dependent iterative workflow, more trustworthy than a preregistered model that has not been tested at all.

2. Eliciting domain expertise should probably involve project leadership. But how ?

3. I’m flattered to be considered by Johannes to be a trustworthy researcher. I also prefer publication procedures that rely less on authors’ reputation and more on methods clarity.

Andrew thought:

It’s funny that they characterize us as being “independent,” given that both of us had Earth Institute associations at the time we did the project.

Postdoctoral opportunity with Sarah Cowan and Jennifer Hill: causal inference for Universal Basic Income (UBI)

See below from Sarah Cowan:

I write to announce the launch of the Cash Transfer Lab. Our mission is to build an evidence base regarding cash transfer policies like a Universal Basic Income. We answer the fundamental questions of how a Universal Basic Income policy would transform American families, communities and economies. The first major initiative is to study the effects of the permanent fund dividend (PFD), a policy in Alaska that has paid every Alaskan resident a substantial amount of cash annually since 1982. This is the closest policy in the world to a Universal Basic Income.
You can read about the Cash Transfer Lab here.
We are hiring a postdoctoral associate to work on causal inference modeling. This scholar will work primarily with myself and Jennifer Hill. Details on the position and how to apply is here. We are looking for someone who has expertise in causal inference and has earned a PhD or will by September 1, 2021.
Please spread the word!

Come work with me and David Shor !

Open positions on our progressive data team:

Machine Learning engineer – https://grnh.se/6713732b4us
Software Engineer – https://grnh.se/15ee5a2e4us
Devops – https://grnh.se/aa2bef714us

We are a diverse team of engineers, data scientists, statisticians, and political insiders who are closely connected to some of the most important decision makers in the progressive ecosystem. We worked with central players to develop strategy and direct hundreds of millions of dollars of resources.

Our problems are not well suited to plug and play machine learning solutions – we’re looking for folks to help us think creatively and critically about how best to construct models on tricky datasets, carefully diagnose when they fail, and design and implement new features in our custom Bayesian library.

 

Data For Progress’s RuPaul-Predict-a-Looza

Data for Progress launched the RuPaul-Predict-a-Looza (and winner), the first ever RuPaul’s Drag Race prediction competition. Statistical models versus NYC Council Speaker Corey Johnson. The prize: bragging rights and the ability to add one policy question on the next Data for Progress survey.

First predictions are due this Thursday (February 28). I made a notebook with the rules and a simple model in Stan.

I talked with Ben (Goodrich) and Michael (Betancourt) about possible improvements for folks to try:

  1. I get divergent transitions with the default adapt_delta = 0.8. Michael’s intuition is that constraining eta for losing to be the -1*eta for winning is “manifesting in a little bit of misfit which induces some nastiness in posterior geometry.” Ben suggested “scaling eta by some negative number with an expectation of -1.”
  2. Michael also notes that we lose information by only looking at winners and losers and not the full ranking. He suggests doing an ordinal regression, though the number of categories would be changing.
  3. The easiest improvement is adding more covariates.

NOTE: If you aren’t ready by this Thursday (February 28) you can join next week! If a team does not submit for an episode, it counts as an incorrect prediction. Since early episodes have many contestants and little information for prediction, most teams are likely to get them wrong anyhow. So folks can catch up in later weeks.

We Count! The 2020 Census

Like many readers of this blog, I’m a statistician who works with Census Bureau data to answer policy questions. So I’ve been following the controversy surrounding the added citizenship question.

Andy thought I should write an article for a wider audience, so I published a short piece in The Indypendent. But much more discussion could happen here:

  1. How are citizenship data used in Voting Rights Act cases? For redistricting cases, American Community Survey citizenship data are either imprecise (1 year only) or out-of-date (aggregated over time).
  2. Should we use administrative data? How? 2020 will be the first census to use administrative data for nonresponse follow-up.
  3. Should we adjust for undercount? How? In 1999, the Supreme Court ruled against using adjusted counts for apportionment. See some of Andy’s thoughts here.

Whole books have been written on census politics. Given the stakes, it deserves our attention.

Thoughts? Continue reading

mysterious shiny things

(Disclaimer: I’m new to Shiny, and blog posts, but I know something about geography.)  In the Shiny gallery, take a look at 2001 versus 2002. Something funny happens to Switzerland (and other European countries), in terms of the legend, it moves from Europe to the Middle East. Also, the legend color scheme switches.

 

2001_Shiny

2002_Shiny

 

To reproduce it yourself: download ui.R, server.R, and healthexp.Rds

Have a folder called “App-Health-Exp” in your working directory, with ui.R and server.R in the “App-Health-Exp” folder. Have the dataset healthexp.Rds in your working directory.
Then run this code:

if (!require(devtools))
  install.packages("devtools")
devtools::install_github("jcheng5/googleCharts")

install.packages("dplyr")
install.packages("shiny")

library(shiny)
library(googleCharts)
library(dplyr)

data = readRDS("healthexp.Rds")
head(data)

# Problem isn't the data, it seems that Switzerland is in Europe 
# in both 2001 and 2002:
data[data$Year == 2001 & data$Country == "Switzerland",]
data[data$Year == 2002 & data$Country == "Switzerland",]

runApp("App-Health-Exp")

Anyone know what is happening?