I Should Never Have Recommended The Skidmore Survey

In a previous post I urged the readers of this blog to participate in a Skidmore student’s survey on politics. I regret that I did so prior to viewing the survey.  Having now looked at it I am quite troubled by what I found.

First of all, the questions have continually changed.  This is quite bizarre. 

Here are three screen shots of one of the questions on the site on different days. You will see that on each occasion the question morphed.

Example #1

The second question on this page is about the proposed new charter.  In this case it contains the sentence “In addition to this, supporters of charter reform argue it is a government that will bring fresh blood into politics and rid city hall of political cronies by imposing term limits.” 

Example #2

The first question on this page is again about the proposed new charter.  It does not have the business about cronyism and now ends with the question, “If Mayor Joanne Yepsen supports charter reform, would you support charter reform?”

Example #3

The third question on this page is about charter change but in this case it has neither the business about cronyism nor the question about the mayor’s support.

So, having established that the survey lacks the kind of consistency that is required to tally answers, let’s scrutinize the design of the questions themselves.

Let’s take one of the versions of the question related to the charter:

“Every 10 years the mayor of Saratoga Springs appoints a charter review commission in accordance with the city charter. This commission has revised the charter that will result in a more effective, less polarized, and a more/modern [sic] and professional system of government. If Mayor Joanne Yepsen supports charter reform, would you support charter reform?”

This question violates the very basics of what is considered good polling.  To begin with it has as its assumption that the charter as currently proposed “will result in a more effective, less polarized, and more modern and professional system of government.”  Who, one might ask, would be opposed to such an improvement in our city government if it is as described?  But the question is not even whether you approve or oppose the charter itself because it is qualified by the final sentence which asks, “If Mayor Yepsen supports charter reform, would you support charter reform?”  So even if you support the proposed charter reform, the survey wants to find out whether you would change your mind if you learned (in the most unlikely of circumstances) that the mayor opposed it or if you oppose charter reform whether you would change your mind if you learned that the mayor supported it.

One has to ask, what is the purpose of this question?  Is it to find out how much influence the Mayor has over voters?  If that were the purpose then why not simply ask whether the person taking the survey will decide on their vote on charter reform based on what the Mayor advocates?

In fact this question has the unpleasant characteristics of a push poll.  This is a commonly used political strategy where people are asked (usually by phone) their position on an issue which is presented in an extremely slanted way meant to spin the person taking the poll in a particular direction. The results are then used to claim support for a particular position on an issue. After all the question of whether to change the current form of government is quite controversial.  People of good faith are quite divided on the question.  There are those who would vigorously disagree that the proposed charter change would result in the benefits assumed by the way this question was worded but there is no way to register that opinion on this poll as constructed.

Complicating things further is that the charter has not even been completed yet so who knows what it may or may not do.

There are also two odd questions that appear to be related.  The person taking the survey is asked first to rate the city council: 

“On a scale from 1 to 5, 1 being the lowest 5 being the highest, rate the city council.”

Then they are asked to rate the mayor:

“On a scale from 1 to 5, 1 being the lowest 5 being the highest, rate the mayor.”

Now in the Commission form of government the mayor is a co-equal member of the city council.  Again, I would ask, what is the purpose of this question?  Is this an attempt to rate the mayor’s status among the voters?  Why not ask the person being surveyed to rate the other commissioners?

I am a supporter of the Sustainable Saratoga initiative on affordable housing. Nevertheless I am bothered by another question that appears to be a push poll:

“The Saratoga Springs Housing Task Force is working on an initiative that would create more workforce housing in Saratoga Springs. Proponents of the project argue that it will create more affordable housing for residents. Do you support this initiative?”

Who would be against the city having more affordable housing? What is the likelihood of people taking this poll clicking on the “oppose”  button?  In fact, while I disagree with them, there are a number of interest groups that have expressed opposition to the proposed ordinance who say that they are in favor of affordable housing but oppose this particular method of achieving it but they have no way of registering that opinion in this poll. There is no button to push that says  “I support initiatives for affordable housing but not this initiative.”  Is the goal of the survey to assess support for affordable housing initiatives in Saratoga Springs or to assess support for this particular initiative?

Here is another question from the poll:

“Saratoga schools are now going to implement iReady, an online program that pinpoints where students need to improve to pass state-educational tests was[sic] passed despite a current budget deficit. Do you support this addition to the budget?”

First of all the wording of this question is exceedingly awkward to say the least. More importantly, though,  I would expect that most people who participate in this survey are like me in that they have little idea of the actual merits/effectiveness of the iReady program.  Again, the question asserts that it “pinpoints where students need to improve…”  Does it actually effectively “pinpoint where students need to improve”?  We have only the assertion of the person who crafted this survey that it does.  And how expensive is it? How much does it contribute to the deficit? This question is flawed  because it assumes a level of knowledge of survey participants which is probably unrealistic thus results will be unreliable.

Finally, there is the basic problem that this is a self-selecting survey.  That is to say that rather than do a random sample that would be demographically representative of the community, this survey is being filled out by people who simply are interested in responding for whatever reason. They may or may not be representative of the broader community. I again ask, what is the purpose of such a study? What will the results tell us?

I am sorry to be so critical of this survey because I regret that my blog might embarrass the student who is conducting this survey.  My criticism is really directed to whomever the faculty members are who are overseeing the student’s work.