Kevin Esterling, who is involved in evaluating the America Speaks forums, sends along the following (all links added by me):
bq. Archon Fung, Taeku Lee and I are part of a research team that is evaluating the AmericaSpeaks townhall deliberative events on “Our Budget, Our Economy,” held on June 26. The three of us share an academic interest in deliberative field experiments. We were involved to a limited extent in the design of the events themselves. For example, at our request, AmericaSpeaks randomized seating assignments for the 300 or so small group discussions at the 19 events, which may allow us to estimate the causal effects of deliberation as the composition of participants and the nature of deliberation vary across discussion tables.
bq. In earlier postings, John Sides and Andrew Gelman summarize and comment on a paper by Ben Page and Larry Jacobs that expressed several points of skepticism regarding these events and more generally deliberative democracy in practice. A number of their points are empirical, such as the representativeness of the participants who attended and the quality of the deliberation at the event, points we’ll be able to elaborate on in our evaluation of the events.
bq. But what appears to be the greatest concern for the Sides and Gelman posts (and perhaps a motivation underlying the original Page/Jacobs paper) is a suspicion that, since the Peterson Foundation is one among several funders, the events must have had a right-leaning bias. (I should mention too that the funding for our evaluation is from MacArthur, not Peterson.). And if this bias were true, the events would tap into a serious concern among political scientists regarding deliberation, that ordinary citizens are simply malleable in their opinions and attitudes, and biases in the information provided or in the agenda must turn the events a democratic sham; in this case, a Pete Peterson ploy to justify his agenda to dismantle Social Security.
bq. I can say that AmericaSpeaks tried very hard to ensure the information presented and the agenda were balanced and informative, and they structured the events in a way to promote free and open discussion. For example, AmericaSpeaks had the reading materials extensively vetted by 30 organizations, both left and right, and went to considerable effort to do so. And the event also was heavily funded by MacArthur and Kellogg; if the events really were an enormous push-poll engineered by Peterson, it’s not clear why these other two centrist funders would have found these events to be a worthwhile investment.
bq. While AmericaSpeaks’ intent was to be neutral, and I can’t think of anything more they could have done, one certainly can read the materials and debate the extent to which they succeeded. But even if AmericaSpeaks did not succeed in correcting biases in the information, or even if the Peterson Foundation indeed had hijacked the event for its purposes, it is still conceivable that the 3,500 ordinary citizens who attended the events, who spent an entire Saturday reading through dense material and discussing the budget and fiscal policy with total strangers, strangers who were often very different from themselves, working without a lunch break (and in my case, on an absolutely beautiful day stuck in the Federal Circuit courthouse in Pasadena), perhaps these ordinary citizens were able retain a capacity to be critical and to think for themselves within the situation.
bq. There is growing experimental evidence that ordinary citizens have considerable capacity to become informed and to think critically when given a motive and an opportunity to do so, that is, when the deliberative exercise is well structured. For example, in some of Jamie Druckman’s experimental work (pdf), question wording framing effects disappear in mixed groups but not homogeneous groups; Phillip Tetlock’s experiments show certain forms of accountability increase cognitive integrative complexity; some of my own work with Mike Neblo and David Lazer (pdf) shows that participants in a deliberative exercise more deeply encode policy information. Deliberative events, that is, can be badly designed, but it is possible for them to be well designed, to foster constructive and informed dialogue among interested citizens, even among those of the ordinary variety.
bq. Was the June 26 event well designed? We don’t even have the data entered yet so it’s too early to tell. But we do have one set of findings to suggest that they were simply not Peterson-engineered sham events. As John Sides points out in his posting, the opinions that citizens registered at the event closely reflect survey research findings on public opinion on fiscal issues. This finding contradicts all three premises of the Page and Jacobs paper, and the Sides and Gelman posts: these findings are simply inconsistent with the assertion that the events were badly unrepresentative, biased, or full of citizens who were easily manipulated.
bq. Sides concludes that the events taught us nothing about public opinion, but in fact we believe the evaluation will tell us a great deal. First, even if the aggregate opinion at the events mirrors survey research findings, there is some value in finding that considered opinion is consistent with general-population surveys, and indeed helps to show their validity. And while much of political science is focused on preferences and their tendency to change in response to deliberation, we believe there is considerable value in understanding how citizens respond to the events themselves, and in particular if citizens can learn to appreciate the positions and rationales of those with whom they disagree; if deliberation increases citizens’ motivation to become involved in politics; whether citizens change in their beliefs regarding legitimacy of a political process that values citizen input, and so on. That’s the focus of our research.
bq. We would ask Page and Jacobs to give us time to finish the evaluation before reaching conclusions regarding the deliberative quality of the events. For anyone interested, you can sign up for working papers by sending an email to fiscalfutureresearch@ash.harvard.edu.
bq. And perhaps of even greater value, those who value deliberation on normative grounds but are skeptical of deliberative democracy in practice might do well to examine the structure of the events, and to propose improvements. AmericaSpeaks’ single objective is to improve the discourse in our civic life, something that is sorely needed today, and they would love to hear any suggestions for how to improve on their design and methods.
See also Archon Fung’s response.