As It Turns Out It Wasn’t A Real Survey At All

Bob Turner, Skidmore Government professor and chair of the Saratoga Charter Review Commission, asked me to solicit participation in a survey   one of his students  was working on for  a project for a statistics class.  Last Tuesday night I discovered that the question regarding the proposed Saratoga charter in the Skidmore student’s survey was worded as if the survey were a push poll. I immediately emailed Bob Turner.  To his credit, he emailed me back late that night with his cell phone number and encouraged me to call him the next morning which I did.

His first words to me were, “you discovered it’s not a survey.”  Actually, I hadn’t yet.  He told me he had had no involvement in drafting the questions.  He explained, though, that there was only one question that the project was really interested in.  All the other questions were “filler.”  The point of the exercise was to determine what would be the most effective way to word a particular question to get a favorable response.   As it turned out, the topic of the question to be studied was the proposed city charter.

I was told the student is interested in being involved in political campaigns.  This exercise was meant to master a technique that is being used by political strategists to determine how to present a topic in a way that will gain support from voters.  Once the most effective wording is determined it can be incorporated into campaign materials.

So while participants in this survey were  led to believe that the survey truly sought to determine how the public felt about certain public policy issues, this was simply not the case.

Mr. Mann writes, “This project is investigating the effects of framing on political attitudes about local issues.” Later he writes, “Perhaps the best known example today is the way adding the President’s name to a policy changes support for that policy….”

More troubling to me was the following, “The Saratoga Springs survey randomly assigned people to see different versions of questions about local issues.”  I leave it to readers to make up their own minds but I believe he was trying to hide the fact that there was really only one question (not questions) that the project was concerned about and that was the question about the proposed city charter. In fact, both based on Bob Turner’s comments and my review of the pages from different surveys, there was only one issue/question that differed over the four “surveys” I was able to compare.  This was the question about the charter.


From Wikipedia on Push Polls:

“A push poll is an interactive marketing technique, most commonly employed during political campaigning, in which an individual or organization attempts to manipulate or alter prospective voters’ views/beliefs under the guise of conducting an opinion poll.

In a push poll, large numbers of voters are contacted with little effort made to actually collect and analyze voters’ response data. Instead, the push poll is a form of telemarketing-based propaganda and rumor mongering, masquerading as an opinion poll.”


Consider the version of the question on the charter that asserts that it will save the city $500,000.00. One wonders where the student or Mr. Mann came up with this number.  The charter is not complete and no financial analysis has been done by the Charter Commission yet.  Push polls depend upon people who take these push polls spreading the “message.” One can imagine someone who took the survey hanging out at work at the coffee machine the next day saying, “I understand the proposed charter will save us half a million dollars.”

Here is the page.  Check the question at the top:

5000page

Given that the question of whether the city should adopt a new charter is so controversial, could Mr. Mann who, after all is a political scientist who specializes in election polling, be so unaware of the potential impact this poll might have?

It is also important to realize how widely distributed this survey was.  This appeared on the Times Union website:

Midschoolportal

This appeared on the portal used by the Saratoga Springs Middle School and another similar one was posted on the High School portal.  I have no idea where else they may have put it.

FullSizeRender (2)

 

The Morality Of The Project

Mr. Mann notes in his comment that:

“Each student project was reviewed and approved by the Skidmore College Institutional Review Board based on federal and state research ethics rules and guidelines.”

I have no question that this project broke no laws.  I find it quite credible that it met the standards of the review board and federal and state codes he references.  I have no question that this is a clever tool that can be employed in political campaigns and that a student who mastered this might be very attractive to a politician putting together a paid campaign team.

Having granted all that, asking people to volunteer to assist in a student project by misrepresenting the true purpose of what they are assisting in is something that offends me personally. I apologize profusely to the people who may have responded to this survey as a result of my solicitation of their help on this blog.  I must say that Bob Turner, when I pointed out my concerns, was generous enough to apologize to me.

Fundamentally, this project took advantage of people’s good will. The student and her adviser exploited the generosity of people in our community who thought they were supporting a project meant to provide insights into how this community viewed important issues. The student and adviser cynically had no interest in the answers to most of the questions they were asking. They simply wanted to master a technique on how to craft a message to maximize how to sell something. In this case it was how best to sell the idea of a proposed charter.

 

A Skidmore College Professor Defends The Survey That Wasn’t A Survey

[JK: I received a comment from Chris Mann who was identified as the “investigator” related to the Skidmore Student Survey.  Apparently this means that he supervised the student who wrote it.

Mr. Mann is an Associate Professor in the Political Science Department of Skidmore College.

If readers have some difficulty reading Mr. Mann’s text it is probably because of his writing style.  It took me several readings to feel that I had grasped what he was trying to say. There is a regrettable trend in academia to write unnecessarily dense prose. Having been a doctoral candidate in history in another life, this is not the first time I have encountered writing like this. Often this style is meant meant to intimidate and impress the reader.  Sometimes it is deliberately obtuse to either hide the banality of what is being said or to obscure some point.  I will be “desconstructing” [that is a word they use] his comments in the blog immediately following this one.  A clearer way of putting it is that I will attempt to make what he has written easier to understand.

I received a subsequent email from Mr. Mann asking when I expected to post his comments.  The next morning I wrote to him explaining that I was in Boston and that I expected to be back in Saratoga later that day and expected to draft a response to his statement and publish both his comment and my response then.

Mr. Mann was most unhappy with this delay.  Below, following his original comment, is his email to me in which he shares his frustration with me for my “embargo” of his comment.

It turns out that what was presented as being a survey was not a survey at all.  I will be explaining this in the blog that follows and discussing some of the difficulties I have with Mr. Mann’s defense.]


 

April 27, 2017 5:42 PM

As the Skidmore faculty member supervising the survey project discussed in this blog post, let me clarify the nature of this survey, a final project for a student in my research methods course (Skidmore course PL367) on how randomized experiments, which are more familiar in medical and drug safety trials, can be used in political science. Each student is conducting a project using randomized trials, although this is the only public opinion survey in the Saratoga Springs area. Each student project was reviewed and approved by the Skidmore College Institutional Review Board based on federal and state research ethics rules and guidelines.

This project is investigating the effects of framing on political attitudes about local issues. Framing has been widely studied in the social sciences since the groundbreaking (and eventually Nobel Prize winning) work of Prof. Daniel Kahneman and Amos Tversky. For an interesting and accessible account of the origins of this area of research, I suggest Michael Lewis’s recent book “The Undoing Project”. Framing is routinely studied in political polling. Perhaps the best known example today is the way adding the President’s name to a policy changes support for that policy (e.g. public opinion about Obamacare is significantly different than for the Affordable Care Act, even though both names refer to the same policy). The Saratoga Springs survey randomly assigned people to see different versions of questions about local issues, as noted in the blog post above. The purpose of the project is to assess how these randomly assigned frames change public support. Each framing statement was crafted to reflect a possible framing by supporters of the policy. A larger project would have explored opposition frames about policies as well.

The blog post correctly noted that the project is not using a representative sample of the Saratoga Springs. In a world of unlimited resources, this project would have used a representative sample but Skidmore does not have these kinds of resources for every student course project. We believe that the local political activists recruited via a variety of social networks still provide relevant insights into attitudes about local political issues.

Unfortunately, data collection for this project has been stopped because this blog post described key aspects of research design, rendering any subsequent data invalid. The student is completing the analysis with data collected prior to the blog post. These results will be available to anyone who is interested.

On behalf of the student and myself, I want to thank all of those who took the time to participate in the survey.

For anyone with questions or concerns, the final page of the survey offered contact information for the student, me, the chair of the Skidmore Political Science Department, and the chair of the Skidmore Institutional Review Board. As of this writing, none of us has been contacted by phone or email with any questions or concerns. A handful of comments about the survey were posted on social media where participation in the survey was requested.


April 28, 2017 8:29 AM

Mr. Kaufman,

I provided a comment to your blog yesterday evening with extensive details directly related to your posting. You acknowledged receipt of this comment in an email to me within moments of this posting. I am inquiring about your comment moderation policy and whether there is something about the information about my student’s research that has led to my comment not being approved. Please let me know the standards for comment approval, as the only statement I could find was your important statement about civility and truthfulness that I wish were more widespread online.

Your email indicated that you have additional questions about the research project and I would like to share the answers with you and your readers. As indicated in the final page of the survey you have written about and my as-yet-unapproved comment, I and others at Skidmore are available to answer questions and address concerns from the beginning of the project through sharing results after the analysis is completed. However, it is difficult to fulfill this commitment when my communication is not approved to be shared with your readers after you raised a set of questions and concerns. Now that you have started a conversation in this public forum, that is the appropriate venue for the conversation to continue so that your readers can be fully informed about the research project. I look forward to a continuing discussion about the project on your blog so that I can answer all of your questions.

Thank you for your interest in the research conducted by my student. I look forward to continuing the discussion that you started on your blog.

Respectfully yours,

Christopher Mann

————————————————————————

Christopher B. Mann, Ph.D. Assistant Professor of Political Science

Skidmore College Department of Political Science

Ladd Hall 310

815 North Broadway

Saratoga Springs, NY 12866

cmann@skidmore.edu www.christopherbmann.com

 

I Should Never Have Recommended The Skidmore Survey

In a previous post I urged the readers of this blog to participate in a Skidmore student’s survey on politics. I regret that I did so prior to viewing the survey.  Having now looked at it I am quite troubled by what I found.

First of all, the questions have continually changed.  This is quite bizarre. 

Here are three screen shots of one of the questions on the site on different days. You will see that on each occasion the question morphed.

Example #1

IMG_0479 (4)The second question on this page is about the proposed new charter.  In this case it contains the sentence “In addition to this, supporters of charter reform argue it is a government that will bring fresh blood into politics and rid city hall of political cronies by imposing term limits.” 

Example #2

IMG_0477 (3)The first question on this page is again about the proposed new charter.  It does not have the business about cronyism and now ends with the question, “If Mayor Joanne Yepsen supports charter reform, would you support charter reform?”

Example #3

IMG_0476 (3)The third question on this page is about charter change but in this case it has neither the business about cronyism nor the question about the mayor’s support.

So, having established that the survey lacks the kind of consistency that is required to tally answers, let’s scrutinize the design of the questions themselves.

Let’s take one of the versions of the question related to the charter:

“Every 10 years the mayor of Saratoga Springs appoints a charter review commission in accordance with the city charter. This commission has revised the charter that will result in a more effective, less polarized, and a more/modern [sic] and professional system of government. If Mayor Joanne Yepsen supports charter reform, would you support charter reform?”

This question violates the very basics of what is considered good polling.  To begin with it has as its assumption that the charter as currently proposed “will result in a more effective, less polarized, and more modern and professional system of government.”  Who, one might ask, would be opposed to such an improvement in our city government if it is as described?  But the question is not even whether you approve or oppose the charter itself because it is qualified by the final sentence which asks, “If Mayor Yepsen supports charter reform, would you support charter reform?”  So even if you support the proposed charter reform, the survey wants to find out whether you would change your mind if you learned (in the most unlikely of circumstances) that the mayor opposed it or if you oppose charter reform whether you would change your mind if you learned that the mayor supported it.

One has to ask, what is the purpose of this question?  Is it to find out how much influence the Mayor has over voters?  If that were the purpose then why not simply ask whether the person taking the survey will decide on their vote on charter reform based on what the Mayor advocates?

In fact this question has the unpleasant characteristics of a push poll.  This is a commonly used political strategy where people are asked (usually by phone) their position on an issue which is presented in an extremely slanted way meant to spin the person taking the poll in a particular direction. The results are then used to claim support for a particular position on an issue. After all the question of whether to change the current form of government is quite controversial.  People of good faith are quite divided on the question.  There are those who would vigorously disagree that the proposed charter change would result in the benefits assumed by the way this question was worded but there is no way to register that opinion on this poll as constructed.

Complicating things further is that the charter has not even been completed yet so who knows what it may or may not do.

There are also two odd questions that appear to be related.  The person taking the survey is asked first to rate the city council

“On a scale from 1 to 5, 1 being the lowest 5 being the highest, rate the city council.”

Then they are asked to rate the mayor:

“On a scale from 1 to 5, 1 being the lowest 5 being the highest, rate the mayor.”

Now in the Commission form of government the mayor is a co-equal member of the city council.  Again, I would ask, what is the purpose of this question?  Is this an attempt to rate the mayor’s status among the voters?  Why not ask the person being surveyed to rate the other commissioners?

I am a supporter of the Sustainable Saratoga initiative on affordable housing. Nevertheless I am bothered by another question that appears to be a push poll:

“The Saratoga Springs Housing Task Force is working on an initiative that would create more workforce housing in Saratoga Springs. Proponents of the project argue that it will create more affordable housing for residents. Do you support this initiative?”

Who would be against the city having more affordable housing? What is the likelihood of people taking this poll clicking on the “oppose”  button?  In fact, while I disagree with them, there are a number of interest groups that have expressed opposition to the proposed ordinance who say that they are in favor of affordable housing but oppose this particular method of achieving it but they have no way of registering that opinion in this poll. There is no button to push that says  “I support initiatives for affordable housing but not this initiative.”  Is the goal of the survey to assess support for affordable housing initiatives in Saratoga Springs or to assess support for this particular initiative?

Here is another question from the poll:

“Saratoga schools are now going to implement iReady, an online program that pinpoints where students need to improve to pass state-educational tests was[sic] passed despite a current budget deficit. Do you support this addition to the budget?”

First of all the wording of this question is exceedingly awkward to say the least. More importantly, though,  I would expect that most people who participate in this survey are like me in that they have little idea of the actual merits/effectiveness of the iReady program.  Again, the question asserts that it “pinpoints where students need to improve…”  Does it actually effectively “pinpoint where students need to improve?  We have only the assertion of the person who crafted this survey that it does.  And how expensive is it? How much does it contribute to the deficit? This question is flawed  because it assumes a level of knowledge of survey participants which is probably unrealistic thus results will be unreliable.

Finally, there is the basic problem that this is a self-selecting survey.  That is to say that rather than do a random sample that would be demographically representative of the community, this survey is being filled out by people who simply are interested in responding for whatever reason. They may or may not be representative of the broader community. I again ask, what is the purpose of such a study? What will the results tell us?

I am sorry to be so critical of this survey because I regret that my blog might embarrass the student who is conducting this survey.  My criticism is really directed to whomever the faculty members are who are overseeing the students work.

 

 

Correction On Song Post!

Yup, your well meaning blogger managed multiple mistakes in one post.  I think this is a new record.

Rockabilly Song was by Tom Denny and Russ Pittinger.  The other song was written and performed by Tom Denny.  My deepest apologies to Mr. Denny who is also the manager of the tree program.  Go Tom Denny!

Also, Rick Fenton was a forester and not an arborist.

From the website “Under the Oak Tree”

Forester vs Arborist – What’s the Difference?

 Simply put, a Forester will care for the forest as a whole while an Arborist will care for an individual tree.

Foresters will receive their formal education and training in college. The education and training of an Arborist is less formal than a college degree. The International Society of Arboriculture  offers different certification opportunities that are recognized as the gold standard of arborist education and training.

Help Out With Skidmore Survey On Local Politics

[JK:I received the following email from Bob Turner]

Hi, 

One of our Political Science students is doing a short survey about Saratoga Springs politics.  I am trying to help her get as large and representative sample as possible.  It is for a statistics class assignment.  Would you be willing to post this message and link on your blog?   Happy to have her share the results.

 Thanks

Bob

 Survey on Saratoga Springs Political Issues

 Please take just a few minutes to complete a short survey about Saratoga Springs politics. Your participation will provide important insights about current issues in Saratoga Springs. Your participation is voluntary and all responses are anonymous. Click on the link below to take the survey. Thank you in advance for your time.

 Sarina Musallam

 lick on the link below to take the survey.

https://skidmore.qualtrics.com/jfe/form/SV_0HsJAwOzEgxbArP

 

Bob Turner

Associate Professor of Political Science and Environmental Studies and Sciences

Director, Environmental Studies and Sciences Program

Director, Faculty Student Summer Research Program

Skidmore College

Saratoga Springs, NY 12866