The Student Room Group

When surveys go bad and statistics fail - Awkward Channel 4 attitudes poll...

The headline on Channel 4 news:

http://www.channel4.com/news/muslim-islam-britain-population-perception-survey

Londoners generally underestimate the size of the capital's Muslim community, but across the rest of the UK's regional cities, the tendency is for overestimation, a survey by Channel 4 News has found.


Oh no, those nasty rascist[sic] Brits, all those Daily Mail types with their paranoia. Pretty clear what the survey was hoping to find. Below, in the smaller print:

From more than 2,400 respondents, the results showed that people are generally correct in estimating the size of their local Muslim community.


Wow - as many as 2,400 self-selecting respondents, "generally correct". Well if that's not a statistically significant sample, I don't know what is. Let's read on:

Out of all 2,413 respondents, around 29 per cent guessed that the Muslim population in their local city was smaller than it actually was, compared with 32 per cent who overestimated.


Yeah, a definite tendency to overestimate. Isn't that actually called "margin of error"?

How badly are these nasty right-wing fascist neo-Nazi scum overestimating?

Nearly half (49.3 per cent) of all respondents in London guessed that the capital's Muslim population was less than 10 per cent, when it is in fact 12.4 per cent.

More than 60 per cent of respondents in Bradford, where Muslims account for 24.7 per cent of the population, guessed that the Islamic community made up more than 25 per cent of the population.


Oh, the inhumanity! 60% of the 79 respondents to the Bradford question were out by 0.3%. Where will the racism* end?!?

OK, enough of the sarcasm. The raw data spreadsheet is even worse - https://docs.google.com/spreadsheet/ccc?key=0AsCgBt4b6m7UdGlQQjVodm4tdmpZRXZjUVJRTWZLN3c&usp=sharing#gid=0 - you can see why it was written up in such an odd way, now.

I'm pretty sure you won't be seeing much being made of the results of the survey, and I'm pretty sure it failed miserably to do what was hoped of it.

(Note: I do, of course, know that Muslim isn't a race - I'm just mocking the kind of moronic comments the comment thread on Facebook has attracted!)
(edited 10 years ago)
Original post by digitaltoast
The headline on Channel 4 news:



Feel sorry for them.

They paid good money for a survey. It wouldn't have mattered to them if people had underestimated or overestimated the number or got it spot on. They could have made a story out of any of these. However how does:



Extra! Extra! Read all about it! Survey inconclusive!


grab you as a headline.
Reply 2
Had another look at the survey, and a closer look at the numbers, too, and it gets even worse! Let's take a few quotes from the summary:

More than 60 per cent of respondents in Bradford, where Muslims account for 24.7 per cent of the population, guessed that the Islamic community made up more than 25 per cent of the population.


But the only groupings available to choose from are: Less than 1% 1-5% 5-10% 10-25% More than 25%

So 0.3% out, but look what it does to the graph:



The numbers for Wolverhampton are based on ... 25 people.

Exeter looks like it's wildly underestimated it, based on the self-selecting 35 people. But look at the spreadsheet and the numbers say

Under 15
Correct 17
Over 4

So more people were correct than over or under, but because it's an "under or over" graph only, it grossly distorts the result. Yes? No?
I'm no analyst or statistician, and maybe the whole thing is perfectly valid. Am I right or wrong here?

BTW, of course, not only might the experience be based on the area the user is within the city being looked at, but the demographics of that area, and the kind of internet user. For example, is a user in a heavily Muslim or non-Muslim area more or less likely to engage with an online poll for a particular media outlet (in particular, the one which broadcast the Muslim call to prayer every day during Ramadan last year). So many variables, so little explained..

Another thing that illustrates why I think badly written, pointless surveys are a Bad Thing - being inconclusive, and the data being presented in a misleading way, everyone can take what they want from it. Here are just a few of the many things being said on Facebook and twitter:

muslim_survey_quoted.png

"West Midlands is most racist then" - er, not how they infer that, given that the survey is about Muslims, not race.

"Almost everyone in Wolverhampton " [and] "most people in Bradford overestimate the size of Muslim population in their area" - perhaps the misleading infographic has done its job?

"Shame on Wolverhampton and Bradford". "The overestimations in Bradford and Wolverhampton are striking" - Yes, striking. Shame on all 20 of them(!)

"I've been studying the holocaust..." - < OK, thats one's clearly a mental. Still got caught by the survey and won 4 thumbs-up...

"We stand shoulder to shoulder irrespective of race or colour" - despite this having nothing to do with race or colour, it got 8 thumbs.

My feeling is that this survey was designed to provoke and appeal to reactionary morons, who aren't really well known for spending the 30 seconds needed to see the facts.
Not saying it was a conspiracy, but I think the survey has done what it was designed to do. And I don't think misleading infographics are the right thing to do with such an inflammatory issue (even though it shouldn't be one).

(As they are all on open, public forums, I've left the names in so you can google and know I've not made them up!)
Reply 3
These surveys being done at all is creepy as hell.

Quick Reply

Latest