The Jumpstart financial literacy survey and the different purposes of tests

Mark Palko comments on the (presumably) well-intentioned but silly Jumpstart test of financial literacy, which was given to 7000 high school seniors Given that, as we heard a few years back, most high school seniors can’t locate Miami on a map of the U.S., you won’t be surprised to hear that they flubbed item after item on this quiz.

But, as Palko points out, the concept is better than the execution:

With the complex, unstable economy, the shift away from traditional pensions and the constant flood of new financial products, financial literacy might be more important now than it has been for decades. You could even make the case for financial illiteracy being a major cause of the economic crisis.

But if the supporters of financial literacy need a good measure of how well we’re doing, they’ll need to find a better instrument than the Jump$tart survey.

The ‘test’ part of the survey consists of thirty-one questions. That’s not very long but that many questions should be sufficient for a tightly focused, well-structured test. Unfortunately the focus of the Jump$tart survey is ridiculously broad, ranging from investments to retirement to credit cards to debt counseling to auto insurance to macroeconomics to really questionable career advice.

Even within the categories the questions have a random, pulled-from-a-hat quality with no apparent effort to prioritize. There are multiple references to credit histories but no mention of credit scores. None of the few questions on credit cards mention teasers or other cases where rates can change on a credit card. There are no questions that refer to charts or tables though the ability to read both is an essential part of financial literacy.

On the individual question level the situation is no better. Most of the questions are either badly written, trivial/irrelevant, open to interpretation, guessable or factually challenged. The test resembles nothing so much as the homework paper a student teacher might turn in when asked to come up with 31 questions on financial literacy.

One of the more ridiculous items on the test is this one, which looks like an item from the final exam in Wally Cleaver’s high-school civics class:

18. Don and Bill work together in the finance department of the same company and earn the same pay. Bill spends his free time taking work-related classes to improve his computer skills; while Don spends his free time socializing with friends and working out at a fitness center. After five years, what is likely to be true?

a.) Don will make more because he is more social.
b.) Don will make more because Bill is likely to be laid off.
c.) Bill will make more money because he is more valuable to his company.
d.) Don and Bill will continue to make the same money.

You’re supposed to pick c.

They get extra credit for hyerprecision in reporting the percentage of respondents who picked each item. (In the above item, it’s 11.5%, 9.8%, 67,9%, and 10.8%.) I’m speaking ironically here. One of the basic principles of statistical literacy is to (almost) always round percentages to the nearest percent. Then again, if political scientists and economists can’t get this right when presenting their regressions, how can we expect anybody else to know better?

What this test really reminds me of is the written test they give you when you get your driver’s license: a mix of obvious questions, trivia, and weird things that look like trick questions, along with some items that actually might be useful.

But . . . we just graded our Intro to Statistics final exams today, and I have to say we’re not much better. Some of our questions seem pretty pointless too. The difference is that we’re using our exam to grade the students, and I have every reason to believe that higher scores on the exam correspond to better understanding of statistics. In contrast, the questions on the quiz above are for . . . what, exactly? I’m not sure.

P.S. My favorite item on the survey is this one:

12. Barbara has just applied for a credit card. She is an 18-year-old high school graduate with few valuable possessions and no credit history. If Barbara is granted a credit card, which of the following is the most likely way that the credit card company will reduce ITS risk?

a.) It will make Barbara’s parents pledge their home to repay Karen’s credit card debt.
b.) It will require Barbara to have both parents co-sign for the card.
c.) It will charge Barbara twice the finance charge rate it charges older cardholders.
d.) It will start Barbara out with a small line of credit to see how she handles the account.

What’s wrong with this item? To start with, I have no idea who “Karen” is (in choice “a”). But there’s one more thing. Check this out:

barbara.png

An 18-year-old named Barbara, huh? What year did they write this question??

15 thoughts on “The Jumpstart financial literacy survey and the different purposes of tests

  1. Some pre-teens I know learn about finance here from Tim and his robot friend Moby. Use it more to motivate topics than the final word on the topic. The (currently free) lesson on mortgages was produced before the bubble burst.

  2. Okay, I'll bite…Why is it a matter of statistical literacy to round percentages to the nearest percent? Sounds…arbitrary–unless it's for aesthetics.

  3. Peter:

    In what sense is it useful to know that "11.5%" of people responded in a particular way? Saying "12%" would be just as good and would reduce the flow of distracting numbers.

  4. Peter, read some of Ehrenberg's work to find references to the research that led him to conclude numbers in tables should be rounded to 2 (not 3 or more!) digits.

    That does bring me to a pet peeve, though: the statement at the bottom of some tables that says, "Numbers may not sum to 100% due to rounding error." Why doesn't the author fix the problem rather than make the reader fix it mentally? I like J's 'rounddist' function (http://www.jsoftware.com/help/user/script_numeric.htm#rounddist) that addresses that directly.

  5. I tend to round percentages to one decimal place and counts to whole numbers but that is because they are usually mixed in tables and it just a way of guiding the eye rather than for accuracy.

  6. 18. Don and Bill …

    e. Workaholic Bill's wife will leave him after he neglects her. Because he's neglected his body as well, he has premature heart disease. Bill has no friends. His company recently outsourced computer functions to Bangalore so he has no job now, either. He does have student loan debt from those courses. Don meets a wonderful woman at the gym, and they two of them go on bike rides to the park with their two children on weekends.

  7. I'm surprised by the framing, locating Miami on a map seems fairly non-vital for most American high school seniors (getting the state wrong would be more problematic). Knowing what part of Florida Miami is located seems to me to be nonessential knowledge for most. This isn't a defense of competence or cultural literacy of most American high school seniors (or most Americans period) -I'm sure the levels are atrocious.

  8. Peter makes a good point. It could be more explicitly grounded in cog sci theory. I think a stronger case could be made for rounding to the nearest tens percentile, or nearest quintile grouping (except that quintile might lose more people in exoticness for people used to percentile norms). 87% and 32% are probably closer to the overprecision of 87.2% and 32.4% than they are to the intuitiveness of 90% and 30%.

    Watchathink, Professor Gelman?

    I have the sinking feeling someone smarter has already worked through this more thoroughly elsewhere.

  9. Keith:

    Yes, I agree that it can be useful to have numbers to lots of decimal places for the purpose of reproducing the original analyses. I've spent frustrating hours on many different projects, struggling to reproduce what had been done before by someone else. The worst case was when someone had computed standard errors by dividing by square root of N (the population size) rather than square root of n (the sample size). It wasn't as obvious as you might think because the whole thing was buried within a stratified sampling design, so there were a lot of ways for them to have screwed up. I knew there was a problem right away–the standard errors were way too small giving the sample size (something like 1 percentage point with a sample of about 150)–but it took me a while to figure out exactly what they'd done.

    Hopefully Anonymous:

    1. I always assumed they couldn't even find Florida, not that they were placing Miami in the location of Jacksonville or whatever. But maybe I'm wrong.

    2. Yes, I often will round numbers to the nearest 5 or 10 percentage points. For example, 70% of American adults support the death penalty for people convicted of murder.

  10. Q.12 is odd indeed, especially since each of the answers could be right under the right circumstances, and I'm not sure the "correct" answer even predominates. But it depends on how debtworthy "Karen" is.

  11. Andrew:
    There are a variety of problems with the questions. They are written badly. The "Which statements are NOT correct" questions judge the test takers test taking skills, and whether they read the questions not their financial literacy. If you look at the numbers, there are many that carelessly missed these. These were consistently the "hardest" questions, despite not having a harder content level. (Remember, these tests don't count towards a grade – if the students don't care, they can just make pictures using the bubble sheet. I did so on occasion for this type of test.)

    Additionally, many are useless information. For #22, regrading insurance, where coverage types are mandated by law, and explained in the brochure when you buy it, why is this essential financial literacy?

    Or question #25: Which type of investment isn't protected by the federal government? The correct answer should be stocks (which was not a choice,) not state bonds; that's an important point only for relatively sophisticated investors, who want to buy individual bonds. Retail investors don't typically do so.

  12. The most interesting aspect of 'financial literacy' is that even though we KNOW kids aren't learning it in school or rarely at home, the stuff that IS taught is pretty lame.

    I've read studies that say financial literacy education doesn't work but that's comparing it math, English or science that's taught every year for years! We compare it to one semester of financial education. Of course it's not going to work!

Comments are closed.