A Skidmore College Professor Defends The Survey That Wasn’t A Survey

[JK: I received a comment from Chris Mann who was identified as the “investigator” related to the Skidmore Student Survey.  Apparently this means that he supervised the student who wrote it.

Mr. Mann is an Associate Professor in the Political Science Department of Skidmore College.

If readers have some difficulty reading Mr. Mann’s text it is probably because of his writing style.  It took me several readings to feel that I had grasped what he was trying to say. There is a regrettable trend in academia to write unnecessarily dense prose. Having been a doctoral candidate in history in another life, this is not the first time I have encountered writing like this. Often this style is meant meant to intimidate and impress the reader.  Sometimes it is deliberately obtuse to either hide the banality of what is being said or to obscure some point.  I will be “desconstructing” [that is a word they use] his comments in the blog immediately following this one.  A clearer way of putting it is that I will attempt to make what he has written easier to understand.

I received a subsequent email from Mr. Mann asking when I expected to post his comments.  The next morning I wrote to him explaining that I was in Boston and that I expected to be back in Saratoga later that day and expected to draft a response to his statement and publish both his comment and my response then.

Mr. Mann was most unhappy with this delay.  Below, following his original comment, is his email to me in which he shares his frustration with me for my “embargo” of his comment.

It turns out that what was presented as being a survey was not a survey at all.  I will be explaining this in the blog that follows and discussing some of the difficulties I have with Mr. Mann’s defense.]

---

 

April 27, 2017 5:42 PM

As the Skidmore faculty member supervising the survey project discussed in this blog post, let me clarify the nature of this survey, a final project for a student in my research methods course (Skidmore course PL367) on how randomized experiments, which are more familiar in medical and drug safety trials, can be used in political science. Each student is conducting a project using randomized trials, although this is the only public opinion survey in the Saratoga Springs area. Each student project was reviewed and approved by the Skidmore College Institutional Review Board based on federal and state research ethics rules and guidelines.

This project is investigating the effects of framing on political attitudes about local issues. Framing has been widely studied in the social sciences since the groundbreaking (and eventually Nobel Prize winning) work of Prof. Daniel Kahneman and Amos Tversky. For an interesting and accessible account of the origins of this area of research, I suggest Michael Lewis’s recent book “The Undoing Project”. Framing is routinely studied in political polling. Perhaps the best known example today is the way adding the President’s name to a policy changes support for that policy (e.g. public opinion about Obamacare is significantly different than for the Affordable Care Act, even though both names refer to the same policy). The Saratoga Springs survey randomly assigned people to see different versions of questions about local issues, as noted in the blog post above. The purpose of the project is to assess how these randomly assigned frames change public support. Each framing statement was crafted to reflect a possible framing by supporters of the policy. A larger project would have explored opposition frames about policies as well.

The blog post correctly noted that the project is not using a representative sample of the Saratoga Springs. In a world of unlimited resources, this project would have used a representative sample but Skidmore does not have these kinds of resources for every student course project. We believe that the local political activists recruited via a variety of social networks still provide relevant insights into attitudes about local political issues.

Unfortunately, data collection for this project has been stopped because this blog post described key aspects of research design, rendering any subsequent data invalid. The student is completing the analysis with data collected prior to the blog post. These results will be available to anyone who is interested.

On behalf of the student and myself, I want to thank all of those who took the time to participate in the survey.

For anyone with questions or concerns, the final page of the survey offered contact information for the student, me, the chair of the Skidmore Political Science Department, and the chair of the Skidmore Institutional Review Board. As of this writing, none of us has been contacted by phone or email with any questions or concerns. A handful of comments about the survey were posted on social media where participation in the survey was requested.

---

April 28, 2017 8:29 AM

Mr. Kaufman,

I provided a comment to your blog yesterday evening with extensive details directly related to your posting. You acknowledged receipt of this comment in an email to me within moments of this posting. I am inquiring about your comment moderation policy and whether there is something about the information about my student’s research that has led to my comment not being approved. Please let me know the standards for comment approval, as the only statement I could find was your important statement about civility and truthfulness that I wish were more widespread online.

Your email indicated that you have additional questions about the research project and I would like to share the answers with you and your readers. As indicated in the final page of the survey you have written about and my as-yet-unapproved comment, I and others at Skidmore are available to answer questions and address concerns from the beginning of the project through sharing results after the analysis is completed. However, it is difficult to fulfill this commitment when my communication is not approved to be shared with your readers after you raised a set of questions and concerns. Now that you have started a conversation in this public forum, that is the appropriate venue for the conversation to continue so that your readers can be fully informed about the research project. I look forward to a continuing discussion about the project on your blog so that I can answer all of your questions.

Thank you for your interest in the research conducted by my student. I look forward to continuing the discussion that you started on your blog.

Respectfully yours,

Christopher Mann

————————————————————————

Christopher B. Mann, Ph.D. Assistant Professor of Political Science

Skidmore College Department of Political Science

Ladd Hall 310

815 North Broadway

Saratoga Springs, NY 12866

cmann@skidmore.edu www.christopherbmann.com