From 2008:

The candy weighing demonstration, or, the unwisdom of crowdsMy favorite statistics demonstration is the one with the bag of candies. I’ve elaborated upon it since including it in the Teaching Statistics book and I thought these tips might be useful to some of you.

PreparationBuy 100 candies of different sizes and shapes and put them in a bag (the plastic bag from the store is fine). Get something like 20 large full-sized candy bars, 20 or 30 little things like mini Snickers bars and mini Peppermint Patties. And then 50 or 60 really little things like tiny Tootsie Rolls, lollipops, and individually-wrapped Life Savers. Count and make sure it’s exactly 100.

You also need a digital kitchen scale that reads out in grams.

Also bring a sealed envelope inside of which is a note (details below). When you get into the room, unobtrusively put the note somewhere, for example between two books on a shelf or behind a window shade.

SetupHold up the back of candy and the scale and write the following on the board:

Each pair of students should:

1. Pull 5 candies out of the bag

2. Weigh the candies

3. Write down the weight

4. Put the candies back in the bag!!

5. Pass the scale and bag to your neighbors

6. Silently multiply the weight of the 5 candies by 20.(And, as Frank Morgan told me once, remember to read aloud everything you write on the board. Don’t write silently.)

The students should work in pairs. Explain that their goal is to estimate the total weight of all the candies in the bag. They can choose their 5 candies using any method–systematic sampling, random sampling, whatever. Whichever pair guesses closest to the true weight. they get the whole bag!

Demonstrate how to zero the scale, give the scale and the bag of candies to a pair of students in the front row, and let them go.

ActionThe demo will proceed silently while the rest of the class proceeds. So do whatever you were going to do in class. Take a look to make sure the scale and bag are moving slowly through the room. After about 30 or 40 minutes, it will reach the back and the students will be done.

At this point, ask the pairs, one at a time, to call out their estimates. Write them on the board. They will be numbers like 3080, 2400, 4340, and so forth. Once all the numbers are written, make a crude histogram (for example, bins from 2000-3000 grams, 3000-4000, 4000-5000, etc.). This represents the sampling distribution of the estimates.

Now call up two students from the class (but not from the same pair) to look at all the estimates. Ask them what their best guess is, having seen this information. As the class if they agree with these two students. Now give the bag to the two students in the front of the room and have them weigh it.

Punch lineThe weight of all 100 candies will be something like 1658. It’s always, always, always lower than

allof the individual guesses on the board. Write this true weight as a vertical bar on the histogram that you’ve drawn. This is a great way to illustrate the concepts of bias and standard error of an estimator.Now call out to the students who are sitting near where you hid the envelope: “Um, uh, what’s that over there . . . is it an envelope??? Really? What’s inside? Could you open it up?” A student opens it and reads out what’s written on the sheet inside: “Your guesses are all too high!”

AftermathNow’s the time to talk about sampling. Large candies are easy to see and to grab, while small candies fall through the gaps between the large ones and end up at the bottom of the bag. You can draw analogies to doing a random sample by going to the mall or by sending out an email survey and seeing who responds. Ask, How could you do a random sample. It won’t be obvious to the students that the way to do a random sample is to number each of the candies from 1 to 100 and pick numbers at random. Also, as noted above, this is an example you can use later in the semester to illustrate bias and standard error.

P.S. My feeling about describing these demos is the same as what Penn and Teller say about why they show audiences how they do their tricks: it’s even cooler when you know how it works.

P.P.S. Remember–it’s crucial that the candies in the bag be of varying sizes, with a few big ones and lots of little ones!

Update (2014):

The comment thread on that one is pretty good too.

How about asking for the shortest interval they’re sure contains the true weight? That’s what they’d need to write down a prior for the weight.

Rather than dictate the terms of the sampling, would a better exercise be to ask groups of students to come up with their own methods for making such an estimate, and finding out which was the most successful? It seems like prescribing a biased sampling method is a bit of trickery …

In the comments to when it was previously posted, you said you might try Phil’s suggestion and have them estimate the whole bag weight. Did you do this? Did it end up with the true value somewhere in the middle like Galton’s wisdom of the crowd, Vox Populi note? If so, were the students impressed (when presumably they opened that envelope)?

Has there been a smart-ass who just surreptitiously weighed the whole bag? Or is the balance too small for that?

I can imagine the weight being lower than most of the guesses. But lower than

allof the individual guesses? Always? That’s counter-intuitive to me.Are there no contrarians? Or innovative kids employing some unique sampling strategy? Or simply making an arithmetic error?

Or some kid that follows your blog? :)

For comparative two group experiments, there is the example of fast rabbits.

Of 10 rabbits 5 are taken from the common cage and given placebo then the last 5 are give a new wonder drug.

The wonder drug is observed to have made the rabbits faster (because they escape being picked first.)

(Maybe not as initially obvious that small candies are better at escaping and so provides more surprising lesson.)

Unfortunately here, the frequency with which this actually happened in actual experiments that were then published is not zero.

A similar example that I use is estimating something like the mean mass of exoplanets. The equivalent of slow rabbits and big chocolate bars are the ones we can observed (more) easily (like the close to the star ‘Jupiters’), while Mars-like and Earth-like exoplanets are the nibble rabbits and tootsie pops! As technology gets better it would be good to track things like the mean mass over time.

I don’t know if this example counts in the same theme but I was reminded of it:

If you try to figure the most vulnerable spot of a WW2 fighter plane (perhaps to decide where to add armor) by noting the bullet marks on planes returning from sorties you’ll probably reinforce the wrong spots.

This reminds me of the spaghetti breaking exercise I do sometimes. Give all the students 1 piece of spaghetti. Tell them to break it “randomly” into three pieces. Then check to see if the three pieces can form a triangle. Calculate the percent. While there are always one or two who can’t, it has never failed for me that I get 85% or more who can make triangles; if it was really random you’d expect 25% pretty easy to show graphically).

Why write the numbers down then make the histogram? Why not take a leaf out of Tukey and make the histogram as you collect the numbers? (am I misremembering Tukey?)

Derek:

I find it helpful to have the actual guesses on the board so each student can see all the other guesses. I typically do this in classes with about 20-50 people, hence 10-25 guesses to write on the board. If I was doing it in a class with 200 students, I’d have to collect the guesses in a more automatic way, then I’d probably display the histogram along with a few of the guesses to give students a sense of the actual numbers.

This is great and I can tell already that I’ll use it forever.

One minor addition that can be considered in large rooms is that with a little preparation you can enter the values into a google document like this:

https://docs.google.com/spreadsheets/d/1vnE-iv0A_RGpfTCFPHqYKN2J6lHENUHCr2tlA9g_LYQ/edit#gid=30479198

and have the histogram build itself as you type them in.

The example itself shown there is a bit different: it is contrasting the number of chips in chips ahoy vs a generic brand of cookies. The gauges there show the mean and standard deviation of each grouping, and the gauges adjust as you type in the numbers. One nice aspect of this is that students get an instinct for how many sample values are needed before the gauges of the mean and sd stop moving.

For data entry:

In a large room, or outside of class, you can have students with computers enter information into a Google Form:

https://docs.google.com/forms/d/1zuWBUuqiSCqcTJnx-JO5AumD9HX595oo7yGMO_Rydug/viewform

to populate a spreadsheet:

https://docs.google.com/spreadsheet/ccc?key=0Av_floYOz0UrdEFmMHZNX25LZXNhMUZrRnItQnhHT3c&usp=drive_web#gid=2

thanks again for this great idea!

Andrew, I wasn’t suggesting not writing the guesses on the board. I was suggesting writing each guess in its bin on the board, and watching the guesses build up a histogram automatically. I believe Tukey called these “semigraphic tables” in the sixties, no computer technology required. The idea in those days was to minimize the expensive use of a statistician’s time by cutting out repetitive actions: write the numbers, get the graph for free.

I went looking for an example of what was in my head, but can’t find one straight off. There’s a “stem and leaf” plot here, which would work, but I was thinking something more familiar to your students. Imagine each guess is handed to you as a wooden block, and you place the block in its bin(don’t bother sorting intra-bin), and watch the distribution build up. Only you’d do this by calling out guesses and writing them on a board instead.

We did this in lecture time last week for a group of about 80 students (first-year Sociology undergrads at City University London) – worked brilliantly, of course – thank you Prof Gelman!

Because we were short on time and large on class size, the way I collected results was to draw the axes for a histogram on the board, with 1000g-wide intervals marked on the x-axis, then called them out in turn and asked students to raise their hand if their estimate fell into the interval that I was pointing at. Not so nuanced, but it was quick and easy, and I think it did the job, so I thought I’d mention it here in case useful.

I can attest that this activity really hits a note with students. I ran into some former students (from a class I taught 5 years ago) at a party recently and the instant the host brought round some dessert, they reminded me to select my piece randomly…

[…] great in-class exercise for introducing statistics students to the concepts of sampling error and bias, and to the […]

[…] The Candy Weighing Demonstration This experiment is a demonstration of the problem with crowdsourced data. Individuals tend to overestimate. Crowdsourcing is plagued by the same problem that makes performance management so difficult: People suck at self-appraisal. […]

This morning I ran this experiment in my core MBA Managerial Statistics class (about 30 students, all working professionals in an evening/weekend MBA program). It worked brilliantly. Thanks for the idea.

> And, as Frank Morgan told me once, …

Frank Morgan? https://www.youtube.com/watch?v=8TC1c364guA

[…] part of my visit I sat in on an intro statistics class and did a demo for them (probably it was the candy weighing but I don’t remember). At that time I picked up an information sheet for the course: […]

We tried this for visiting year 9-10 students, but using plastic dinosaur toys (and some little 1×1 inch laminated cards) instead of candy. They were in an open box instead of a bag and we encouraged the students to think about a representative sample. Even so, only two samples out of about forty were anywhere near the truth.

Quite a few groups thought about a division into small, medium, and large, but they then took the same number from each stratum.

One group actually invented systematic sampling: lining up the dinosaurs and taking an evenly-spaced sample.

A folllowup question: Can the students see all the candies? Is the set of candies in a clear ziploc bag, for example? Or is it a dark bag and they have to stick their hand in and come out with 5? Doing it in a dark bag make it seem very likely they get too many large candies. But doesn’t fit with the instructions to give them the freedom to do a systematic sample, random, etc. If they’re looking at a clear bag, couldn’t they eyeball the number of little things and scale up to the whole population? And do we tell them there are 100 items total? I’ve done this in a Research Design course for several years (and am scheduled to do it again tomorrow), but always feel like I’m missing something.

Jeffrey:

I use an opaque plastic bag, and students can feel free to look in it if they want. I have them do the sampling in pairs, and I don’t give them lots of time on it. So they have time to think about doing a systematic sample but they don’t have time to lay out all 100 candies etc. If any pair of students tries anything too elaborate, I tell them to just take a sample to keep things moving.