Skip to content

Beware of polls, young people edition

There's a reason not to put too much faith in polls. Here's an example from the Harvard Youth Poll, which surveys only people age 18-29:

In the first question, 4% of the respondents say they are transgender or non-binary. In the very next question, a tenth of those who said they are, now say they aren't. An even larger number suddenly prefer not to say even though they had just said.

This is not something complicated about the world, where you might be unsure of the nuances. It's a question about yourself, and within the space of a few seconds a large number of respondents flipped for no apparent reason.

Also, the first two questions suggest that 2% of the respondents consider themselves non-binary. But in the third question 0% say they are non-binary. Come on.

Here's another:

This is why I'm skeptical of some recent polls showing that Biden has lost significant support among the young or the Black or whatever. In the right-hand column, where we eliminate all the "would not vote" folks, 37% say they'll vote for Trump compared to 28% who voted for him in 2020. Does anyone really believe this? I don't.

12 thoughts on “Beware of polls, young people edition

  1. cmayo

    I don't jump to the "they say they're transgender/non-binary and now they don't" conclusion so quickly. I see that 9% and the proper conclusion is not that they're giving conflicting answers on purpose but that the answers are the result of some other factor: either the respondent is not paying attention, answered mistakenly, or the assumptions undergirding the answers to your questions are not accounting for all possibilities.

    As for the last bit, I believe it. The number of young people, very much especially young men, who have fallen victim to right wing rot online is alarmingly high. It's at the point now where I just kind of assume that any young white male I see is cancerously conservative given just a single other social cue in that direction: Virginia "No Step on Snek" license plates, anything pro-gun or even slightly misogynistic, tasteless joking, trucker hat, hunting gear or camo, etc. So much for the young people saving us, and I'm not even that old which makes it worse. I've got to live with it for longer than older non-conservatives do.

  2. BBCWatcher

    Question 159's results are even more inconsistent with Question 143's results. Q159 simply asks about the respondent's gender. Result: 50% male, 50% female, 0% non-binary. Obviously there are some people age 18 to 29 (including in this survey, in Q143 plus Q144) who identify as non-binary. However, for whatever reason(s) Q159 wasn't sensitive enough to pick them up even within the same survey. Is it possible respondents who indicated they're non-binary in Q143/Q144 interpreted Q159 as essentially "If you absolutely must pick male or female (based on your past gender identity for example) which would you choose?"

    1. Coby Beck

      yes. big difference from "do you consider yourself non-binary" and "are you male/female". I can easily see someone answering yes to both, ie "I'm female" (as in that's what the paperwork says and the world sees), but "I consider myself non-binary".

    2. Crissa

      Does it? Because there's no question listed in Kevin's quote.

      Very few people have an ID or birth certificate that says nonbinary.

      Also, I would be suspicious of a poll that found 50/50 male vs female. That's an almost impossible distribution, especially given the US population is 51% female.

      1. Crissa

        That's not even including that a portion of people who find themselves outside of the binary genders find other terms; some of them are cultural like two-spirit or hijra; others are medical like intersex; and yet others identify as gender fluid or feel strongly for a third (or whatever) gender.

        So how you ask these questions matters alot, and this survey doesn't tell us.

  3. zaphod

    "Does anyone really believe this?"

    Yes, I do. Joe Biden is incredibly unpopular. 56% disapproval in the latest average, 39.5 approval.

    The question now is quickly becoming not "will Biden win", but how will we survive another Trump presidency?

    1. Joseph Harbin

      If there is reason to question the validity of election polls, I would think the validity of approval polls may need to be questioned also.

  4. Excitable Boy

    “This is why I'm skeptical of some recent polls showing that Biden has lost significant support among the young or the Black or whatever. In the right-hand column, where we eliminate all the "would not vote" folks, 37% say they'll vote for Trump compared to 28% who voted for him in 2020. Does anyone really believe this? I don't.” - Kevin Drum

    “If the presidential election were held today, President Biden would outperform former President Trump among both registered (50% Biden, 37% Trump) and likely young voters under 30 (56% Biden, 37% Trump). When there is no voter screen (i.e., all young adults 18-29), the race narrows to single digits, 45% for President Biden, 37% for former President Trump, with 16 percent undecided.“

    https://iop.harvard.edu/youth-poll/47th-edition-spring-2024

    These are three different groups. All voters registered and non registered voters are included in question #26 that you have in your post. In that pool of people, Biden leads by 7% among those people when you count those that will not vote and are ineligible. That moves to 8% if the 16% that won’t vote are removed. This pool of people is 1,682 people, because it’s not excluding non registered voters. The question #6 is only including registered voters as of 2020, that number is 1,196. Again these are two completely different groups.

    When they remove the non-registered voters Biden leads by 13%. Then, when registered voters that are unlikely to vote are pulled out of the group, Biden leads by 19% among likely voters. That number is not the 33% lead of the #6 poll, but much closer to it than the poll of all young people.

    I saw a number from a different poll of ALL VOTERS that had Biden up by 18% among “definite voters’” last week.

    These polls are not doing a good job of screening out the 40% of Americans that don’t vote and not focusing enough on the 60% that will make up this year’s electorate. That’s why we need to beware of them.

  5. Joseph Harbin

    From the "why I'm skeptical of some recent polls" dept.:

    Pennsylvania polling, April

    Morning Consult, 4/13: Trump +1
    Beacon, 4/16: Even
    Beacon, 4/16 (inc others): Trump +2
    Kaplan, 4/21: Trump +5
    CBS, 4/25: Trump +1

    Nothing good here for Biden.

    But ... but ... but one week ago, on Tues, 4/23, Pennsylvania held its primary election.

    Biden: 945,712
    Trump: 792,676
    Haley: 158,174
    Phillips: 69,307

    Democrats, Total: 1,075,464
    Republicans, Total: 950,850

    To believe that polls are a true indication of how people will vote, you have to believe that polls are more a reliable indication than the votes that people actually cast. Nonsense.

    Things can change between now and the election in November, but right now Joe Biden is ahead in Pennsylvania despite every April poll showing Trump even or ahead.

  6. Narsham

    I agree with the general point but not the specific example.

    Setting aside that this poll appears to be asking over 100 questions (assuming there aren't lots of branching question lines) and appears to have been handled by "interview," it's producing three different answer sets to three different questions.

    First question: "Are you transgender or nonbinary?" A face-to-face interview conducted by a Harvard undergraduate student isn't a neutral setting, so not everyone may have been willing to respond, but set that aside. 92% said that they were neither transgender or nonbinary, which is straightforward enough, so up to 8% might identify that way whether or not they were comfortable responding.

    Second question, only to those answering Yes to the first, is different because it specifies and breaks down the categories: Male to female, female to male, gender non-binary (ie. neither male nor female identifying), or none of those. Some interviewees may have misunderstood the first question and are now "no, none of those," and I suppose a few might be comfortable answering the first question but not the second. But please recognize the three options are incomplete. What if someone identifies as trans but has not transitioned? In other words, they were assigned female at birth, identify as male, but have not undergone any procedure to transition to becoming male? You aren't nonbinary, because you identify as male. But you also haven't transitioned physically. What about someone partially transitioned to male: if that were you, do you answer as if you'd fully transitioned, or take the none of the above response?

    To be clear: the question is poorly worded, so the answers are going to produce inconsistencies.

    As for the final question, the context is questions about race/ethnicity, then gender, then marital status. These are the last questions asked in the poll and appear to have been used to sort responses. Perhaps the responses reflect "how do you answer this question when confronted with Male/Female only choices" despite the existence of nonbinary as a listed option. But I observe that every single response on the poll cross-tabs is subdivided by two gender categories: male and female. No nonbinary option exists. Is that because no interviewee was willing to answer question 159 as "nonbinary" and they'd have included the third cross-tab had 4% of people said they were? Or did the pollsters not want to subdivide results three ways according to gender and thus prompted this response?

    IMO, pollsters are very good at the sorts of statistical sampling and modeling needed to do the job, but are either remarkably bad at writing questions and at human psychology or are largely interested in influencing the kinds of results they get. I have been offended by how poorly written some recent polls have been.

    1. Crissa

      It also doesn't include the cultural castes and names for non-binary or other genders; such as two-spirit or hajira; nor modern terms like intersex or gender-fluid.

Comments are closed.