Over at Pollster, there’s a bloggy roundtable on Stan Greenberg’s new book. The biggest controversy to emerge is an argument between Stan Greenberg and Mark Penn over their respective roles in the 2004 Tony Blair campaign. Essentially, Greenberg accuses Penn of rigging question and failing to provide the full results of his polls to the Blair campaign.
Penn rejoins:
bq. . The questions in the polls went through a team vetting process and it was up to Labour not me to determine what Stan got and what Stan did not get. I have plenty of letters of transmittal of questionnaires, marginals and crosstabs to the party – they chose what Stan did and did not get – and there were very sensitive questions for small group use only that they decided not to give him.
And Greenberg responds:
bq. Pollsters as a rule share the results for all their questions and hypotheses, even the ones that didn’t pan out. In the Blair campaign, Penn provided a memo with large tables including only the questions he wanted to report; he did not provide a standard book of demographic cross-tabulations. Read Penn’s words carefully, “The campaign received all of the agendas, marginals, as requested without reservation.” In short, he provided breakouts only when asked, in effect keeping his own client and campaign team “out of the loop.”
So in comes Philip Gould, the polling and strategy adviser to Labour from 1987-2005. He attempts to adjudicate:
bq. Greenberg and Penn are very different pollsters, with very different approaches, and crucially with very different value sets, but both have significant contributions to make. Stan puts methodological exactitude first: he is the Volvo of pollsters highly engineered and meticulously thorough. He is strategically astute, but follows the data carefully because he has so much respect for it. His politics are modernising but rooted hard in fairness based populism: his favourite dividing line will always be based on a contrast between the many not the few, his emotional heartland will always remain hardworking families. He is a natural iconoclast, always challenging, often doubting, and leading him sometimes to put flexibility ahead of consistency. Mark also uses and understands data well but leans to strategy ahead of data, and he can be strategically brilliant. He prefers consistency to flexibility, believing that a strategic position once adopted should be held unless there is compelling evidence to the contrary. His instinct is to stick rather than to shift.
He goes on to say this about the distribution of information:
bq. Did Mark Penn make available the ‘agenda’s, marginal and cross-tabs as requested and without reservation’. This is a grey area, but I will try and clear it up.
bq. After a poll Stan normally presented a filled in questionnaire, a full banner book containing complete cross tabs.
bq. Mark had a different approach. Following a poll he quickly made available a full and extensive polling report. This went immediate to the whole campaign. This was not an inconsiderable document. I have one in front of me now: it is 18 pages long; it contains historic voting and favourability data; it closely examines 12 targeting groups ranging from rural lower class Conservatives to union households; it uses seven different batteries to examine campaign issues. It analyses responses to the news and key policy areas. And of course it contains numerous message batteries: in all well over 100 questions were asked and recorded. All of these were analysed by voting preferences, and sometimes by demographic categories.
bq. These reports were extensive and useful documents, far in excess of a normal filled in campaign questionnaire. They did not constitute a full banner book and did not contain ‘full marginal’s’ in the manner favoured by Stan Greenberg, but what Penn did supply was both exhaustive and useful, and certainly met the regular needs of the campaign. As one senior campaign official with responsibility for polling in 2005 has said: ‘Mark Penn ‘could quite fairly argue that the memos were intended for an audience that had no time or interest in delving into every corner of the data. I don’t think that in any way illegitimises the findings or his advice’. On a personal note Mark Penn invariably supplied any additional cross tab or targeting data that I required, and I presume the same is true of others. Two pollsters, two approaches.
Here’s how I read this: Stan Greenberg is right. Penn dispenses what he wants to and withholds the rest — unless it’s specifically requested.
Is this problematic? Gould says no. (Of course, it’s easy to be sanguine when you win!) I say yes. I am not a pollster or a political consultant. But I do analyze polling data for a living, more or less. And if I’m delegating that responsibility to someone else, I most certainly want a “meticulous Volvo” and not someone who “leans to strategy over data” and prefers to stick to a strategic position once it’s adopted. Indeed, this description of Penn makes him sound like someone who bends or at least selects data to fit a strategy, and then resists changing it.
It’s much preferable to have a full rendering of the data available as a matter of course, even if Gould or other advisers would not have had the time or inclination to review it after every poll. The simple reason is that no one person should be expected to “see” everything that a poll is saying. Moreover, as any standard book on polling will tell you, the interpretation of polls is as much art as science. No one person will always paint the perfect picture, as it were. And it’s not as if polling data is particle physics. Readings the text of questions, the marginals, the cross-tabs with demographic attributes — all of that is straightforward, even for a non-specialist.
To me, Greenberg’s M.O. is much closer to “best practices” than Penn’s.