Jumping To Conclusions

Last week, the Pew Research Center for the People and the Press created some big news by releasing their annual survey of world attitudes towards the U.S. as part of the Pew Global Attitudes Project. Eric Alterman, occasionally accused of anti-Semitism for his anti-Israel bias -- and quite defensive about it -- made a big deal of one of its findings:

In the meantime, check out this amazing statistic. "U.S. policies toward the Middle East come under considerable criticism in the new poll. In 20 of 21 populations surveyed - Americans are the only exception - pluralities or majorities believe the United States favors Israel over the Palestinians too much. This opinion is shared in Israel; 47% of Israelis believe that the U.S. favors Israel too much, while 38% say the policy is fair and 11% think the U.S. favors the Palestinians too much." Did everybody get that? The Israelis think we favor Israel too much. Call me an anti-Semite, but I think that makes it true.

Huh? Does it make any sense to believe that half of Israelis think that Americans favor Israel too much? Yes, that is what the poll seems to indicate, and (for a change) the press release accurately reflects what the question asks. But does that sound at all reasonable? It certainly doesn't to me.

There's always a danger of rejecting inconvenient facts just because they don't support our preconceptions, but I don't think that's the case in this instance. A friend of mine likes to quote Carl Sagan's "Extraordinary claims require extraordinary evidence" saying, and I think it applies here. (At least, I think that it's a quote from Sagan.) Given that our recent policy consists of supporting the democratically-elected Israeli government, that "finding" is essentially the equivalent of saying that half of all Israelis are less supportive of Israel than the U.S. And that, despite the fact that the opposition to the current Israeli governing coalition was pummeled in the last election.

That's possible, I suppose -- but does it make sense to say so based only on a single question asked once of 903 people? Isn't it more plausible that (a) the random sample wasn't truly representative for some reason, or (b) there was a translation problem which caused the question to be misinterpreted (We only have the English translation of the questions asked, but presumably the surveys were conducted in the native languages of the various countries), or (c) there was some sort of error in data compilation, or (d) the numbers represent a typographical error, or even (e) fraud occurred?

In this case, I suggest that the design of the question was flawed. The question asked for the respondent's opinion of U.S. policy:

Q.29 What’s your opinion of U.S. policies in the Middle East – would you say they are fair, or do they favor Israel too much, or do they favor the Palestinians too much?

(Again, noting that this is the English language version of the question.) Isn't it plausible that at least some people interpreted "fair" as "equal," and then rejected that option because, clearly, the U.S. does favor one side more than the other? If so, then the only remaining option one could select would be that the U.S. "favors Israel too much." In other words, if one feels that (a) the U.S. supports Israel over the Palestinians and (b) the U.S. is justified in doing so, what does one choose in responding to the question? Neither "fair" nor "favors Israel too much" fully captures that opinion.

We tend to take survey results as gospel, at least within the mathematical confines of the error margin, but they're subject to the same limitations as any other reports: mistakes, lies, confusion, the vagaries of chance. (A 95% confidence interval implies that one out of twenty times, there will be an error larger than the error margin.) We shouldn't ignore data that's strange merely because it's strange, but we shouldn't toss our common sense out the window, either. And if the strange results surprisingly agree with us, we should be doubly cautious that we're not accepting the results merely because they provide validation for our idiosyncratic opinions.