## Question 3 of my final exam for Design and Analysis of Sample Surveys

3. We discussed in class the best currently available method for estimating the proportion of military servicemembers who are gay. What is that method? (Recall the problems with the direct approach: there is no simple way to survey servicemembers at random, nor is it likely that they would answer such a question honestly.)

Solution to question 2

From yesterday:

2. Which of the following are useful goals in a pilot study? (Indicate all that apply.)

(a) You can search for statistical significance, then from that decide what to look for in a confirmatory analysis of your full dataset.

(b) You can see if you find statistical significance in a pre-chosen comparison of interest.

(c) You can examine the direction (positive or negative, even if not statistically significant) of comparisons of interest.

(d) With a small sample size, you cannot hope to learn anything conclusive, but you can get a crude estimate of effect size and standard deviation which will be useful in a power analysis to help you decide how large your full study needs to be.

(e) You can talk with survey respondents and get a sense of how they perceived your questions.

(f) You get a chance to learn about practical difficulties with sampling, nonresponse, and question wording.

(g) You can check if your sample is approximately representative of your population.

Solution: e and f. The purpose of a pilot study is to test out the data collection. The sample size will be too small for a, b, c, d, and g. In some of their earliest work, Kahneman and Tversky documented the common misconception of researchers that data from a small pilot study should closely match the population.

The question would have clearer if I’d inserted the word “small” before “pilot” in the preamble.

1. FMark says:

Do you have the citation for the Kahneman and Tversky paper?

2. Scott says:

If (d)’s wrong how _do_ you decide how large your full study needs to be ?

• Andrew says:

Based on prior information and cost-benefit considerations.

3. I’m with Scott. Where does this Prior Information come from?

• Andrew says:

It comes from scientific knowledge and understanding. Here’s the point: In many cases, for practical reasons you have to decide in advance how large your study will be. Your decision will be based on guesses about the effect size, measurement error, etc. If you don’t have good prior information, your guesses won’t be so great. This is not a controversial idea, nor is it particularly Bayesian. You can see chapter 20 of my book with Jennifer Hill for more on sample size selection.

• Scott says:

That’s fine, but in many other cases you have hardly any notion what the measurement error, say, is going to be, & make it one of the goals of your pilot study to estimate it. It won’t be a very precise estimate, but using the upper 95% confidence bound, say, should be a safe way to plan your big study. (You might need to fudge it to try to take account of differences in sampling strategy too.)

This is in the textbooks too: Ch. 4, ‘Sampling Techniques’, Cochran (1977). Perhaps it depends what kind of studies you mostly do, but I think it’s a mistake to say (d) is simply wrong.

4. Jon Mellon says:

Scale up surveys?

Using snowball methods from random seeds identify a number of gay soldiers. Use a list of names to estimate what proportion of their platoon or other reference group is aware of their sexuality.

Then make your best effort at randomly sampling servicemen (I guess this will probably end up being either stratified or quota sampling). Ask them how many of your [chosen reference group] are gay?

You then apply bayes law to estimate the proportion of gay troops given the probability of knowing a soldier is gay if he is gay and the probability of the “average” soldier reporting that another soldier is gay.

Then you just have the overestimation problem of soldiers incorrectly identifying other soldiers as being gay. So this method will give an upper bound. You could then also include a question on the main survey asking soldiers whether they are gay and this would establish a lower bound.

5. DK says:

Are you absolutely sure that the method you have in mind is the “best currently available method”?

Your question seems to preclude surveys of servicemembers, in which case I would simply derive a correction coefficient between civilians and military members. For this, compare answers to “How likely are you to enlist?” between gays and straights. For gay men the coefficient will be 1.

• Andrew says:

DK:

I was referring to a particular discussion we had in class one day.

6. zbicyclist says:

#3. Same method you would use to estimate the population of zombies, or Nicoles. Then have some way to determine the amount of over/under estimation.

7. DK says:

Umm, something wrong went with my post. I had the last sentence written as “For gay men the coefficient will be 1.”

8. DK says:

OK, so some filter is stripping posts… Annoying! Let’s try it differently: “For gay men the coefficient will be below unity, for lesbians it will be above.”

9. Phil says:

DK, you can’t use “less than” or “greater than” signs, they get interpreted as parts of tags. I find it annoying too. I assume there’s a way to escape them but I don’t know what it is.

Andrew, I guess there’s an implicit assumption that the sample size is really tiny? I’m thinking of d in the context of indoor radon. Radon was “discovered” as an issue in the US when a guy who worked at a nuclear plant kept setting off the radiation alarms at the end of his shift. How was he being contaminated, but nobody else? Eventually they tested him when he arrived an found he was already contaminated, which led them to his house, which had very high indoor radon (and, of course, radon decay products, which were sticking to him and his clothes). One of the early responses was to check a bunch of nearby houses, to try to figure out if this was an isolated problem — maybe the guy had a chunk of radioactive ore in his basement or something. This was a small convenience sample but was still useful for getting an idea of the radon distribution there. Maybe they got the median within a factor of 5 or so.

I guess this wouldn’t meet a strict definitio of a pilot study, but it still seems relevant. If you really have no frickin idea what the distribution looks like, can’t a pilot survey help?

• Andrew says:

Phil:

I agree that your example is important. I don’t think it fits into my categories (a)-(g) above. Let’s add category another possibility:

(h) You can learn something unexpected that was not part of your original design.

• Phil says:

The “pilot survey” I’m talking about here is the informal one in which they made radon measurements in a bunch of houses, just to get a feel for whether it was worth doing a formal survey or was this guy’s house a 4-SD outlier. Getting a rough feel for the distribution wasn’t “unexpected”. But maybe this is a rare case because they just wanted to know some parameters within literally a factor of 4 or 10, and I suppose it’s unusual that you’re that uncertain about the parameters you’re trying to survey.

• DK says:

I am unhappy with your question #2. The answer relies too heavily on the implicit definition of “pilot”. In real world, “pilot” =/= small. Rather, “pilot” = “something close to the max of what we can afford without sacrificing an ability to commit most of our resources in the future”.

• QMS says:

Just to add to the pile, I also think that `d’ is a correct answer in many cases. For example, this is explicitly suggested in the survey guide compiled by Statistics Canada (and, I’ve been told, by the guides put out by the main statistical agency in China). Of course, sometimes it is wrong, if the pilot is *really* tiny.

• Andrew says:

QMS:

Thanks for the info. Could you give the direct quote from the Statistics Canada guide?

• Corey says:

You can use unicode to render inequalities:

– &lt; renders as <
– &le; renders as ≤
– &gt; renders as >
– &ge; renders as ≥

To write a comment about rendering special characters, it helps to know that &amp; renders as &.

10. Jeremy Miles says:

Another approach used in the RAND report “Sexual Orientation and U.S. Military Personnel Policy
An Update of RAND’s 1993 Study” which can be found at: http://www.rand.org/pubs/monographs/MG1056.html [Disclaimer: I work at RAND, and this was done by colleagues of mine).
They estimated the proportion of gays in the military by using the Add Health survey, of around 20,000 high school students who were asked about their sexual orientation when in high school, and then following them to see who joined the military.

From page 101:
“… self-identified gay or bisexual individuals are currently (or have recently been) serving in the military at the same rates as their representation in the U.S. civilian populationâ€”3.7 percent. Among young men who have ever served in the military, 2.2 percent fall into our gay/bisexual category, compared with 3.2 percent of civilian men. In contrast, self-identified lesbian or bisexual women are more common among military personnel than in the civilian population of U.S. young adultsâ€”10.7 percent compared with 4.2 percent.”

• Jeremy Miles says:

Which is remarkably similar to the answer reported by Gates in the answer blog entry.