News Organization PROVES It Doesn't Know Statistics

Okay, so the AP commissioned a study (article here) to look into how younger people digest news. Only one problem. They only looked at 18 people over 6 cities in 3 countries. To pile on nonsense, these groups were ethnically diverse as well.

(Take some time. Try to draw up what this group looks like. I am thinking 3 people per city, 2 cities per country. Now remember, ethnically diverse. So for example, take City A in the United States. 3 people in that city so what? A Black, a white, and a Latino? Maybe one city might have an Asian. Try to fit the aggregate 18 people in 6 cities in 3 countries. Now that you have that in mind, let's move on.)

Okay, now that we have an idea what the sample looks like, how about some statistics. Only one problem. Generally, samples must have 30 members to make the math easier, potentially eliminate outlier types. (For example, sample 30 people out of a population and you might find 20 percent are racist. But what if you only sampled 5 people? We saw that there were 6 of 30 people in the first sample that were racist. What if instead of doing that original sample, we only took the smaller 5 member sample. What happens if 5 of the 6 racists end up in the smaller group?) See, because the AP is trying for diversity, they might end up with 3 or 4, at most 6 people of the same race, but spread out over three countries. Can we really say that a British white person is the same as an American white person? What about an American Asian? Are they the same as an Asian from India?

Many complaints about statistics online, say in political polls, is that 1000 people don't represent the population, even though if done correctly, those polls are accurate to whatever their chosen limit is. But there is no way that someone could reasonably read the conclusions of this AP study and draw anything from it! Imagine if I was in the sample. I barely skim headlines and I don't waste time reading past the first paragraph of news stories that are blatantly biased. But do I represent any of the population? Who knows? That is the point of gathering a large enough sample. I won't argue that a 15 person sample cannot be used because with the right techniques, it can. But, that isn't the case here. A 15 person sample would need to be done for each city and probably for each ethnicity. Here there are 18 people total! We can't break down people into separate groups without ending up with 1 person per group.

Now, to show I actually read to the end in this article, there were 4 U.S. cities, 2 in Britain, and 1 in India. Also, the findings that the researchers did find are valuable; just not scientific. There is nothing wrong with finding that young people tend to do other stuff while reading news (I think that is probably the least controversial finding) and then adjusting news accordingly. But there is no way that this study can prove it with any reasonable level of certainty (most often 95%). If news organizations want to appeal to younger readers, there is nothing wrong with asking random younger readers what changes they want to see. I just don't think this study is really random enough.