One of the key arguments put forward by the Charter Commission for changing the form of government is a “survey” that purports to show an overwhelming desire by city employees to change the form of government. Given my past experience with the declarations of the Commission I secured a copy of the raw data of the survey to take a look at it for myself. What I found was yet again, the extraordinary misuse of data by the Commission in their literature and presentations.
Surveying is a serious business. To the lay person it’s about just asking people some questions. A century of academic study has shown that to produce valid data, the authors must be disinterested and objective. The crafting of questions, the method by which people are selected, and the vehicle by which the targets of the survey are contacted require a vigorous and time consuming process. The reality is that a poorly designed survey provides unreliable results. A survey to be a source of real decision making must address things like how representative are the respondents of the population under consideration? In the case of this “survey”, the respondents were self selected meaning the respondents had to take the initiative to respond, they were not randomly selected. There is no way of knowing therefore to what extent those who chose to respond might be representative of city employees as a whole. Might those who responded, for instance, be those who are particularly unhappy with their jobs or have something else in common. The chair of the charter commission, Bob Turner, admitted that he had no idea who actually got the surveys. In fact, he admitted at a meeting I attended that he did not know how many people were actually surveyed.
We do know that of 398 city employees 75 responded to the survey according to the Commission’s website.
Of the 75 employees, 70 responded to the following statement (#15): I believe city hall would operate better with a full time professional city manager.
The 70 respondents produced the following results:
Strongly Agree 17
Somewhat Agree 18
Neither Agree Nor Disagree 12
Somewhat Disagree 10
Interestingly, unlike the other questions in the survey, for this question they changed the possible responses. In the other survey questions they had:
Neither agree or disagree
Why they chose to change the responses to this question is a mystery to me. It may have just been sloppiness. By the way, the label at the bottom of the chart for those opposed was mislabeled (see chart)
So if you total the first three responses which indicate some kind of support for a city manager you come up with 48. In order to create the greatest contrast, they factored out the people who had no position. This left the 10 who did not think city hall would operate better with a city manager.
Now since these raw numbers were still embarrassingly small, they chose to use percentages Instead. This very much ginned up the impression for the public of their significance. Remember that these are only the percentages of the 70 who responded to this particular question in the survey and not of city employees.
The result is a chart that shows 65.3% pro city manager and 15.3% negative.
So from a small, unscientific sampling—kaboom: the Commission declares the employees of Saratoga Springs overwhelmingly support charter change.
Now I don’t know what the numbers are in city hall regarding feelings toward the two forms of government but neither do the members of the Charter Commission based on this kind of “research.” What I do know is that this should be considered a “questionnaire” rather than a survey. It violates pretty much all the standards of what any serious social scientist would characterize as a survey and it’s results are anecdotal at best and cannot be legitimately used to draw any larger conclusions about what city employees think.
Robert Turner, who chairs the Charter Review Commission is a professor of political science. He was instrumental in creating this “questionnaire” and in reporting its results. For a person who is a social scientist he has both the knowledge and the responsibility to accurately portray what the data produced by this questionnaire represented. If he were not going to characterize it as what it was, a questionnaire, he had the responsibility to include some disclaimer explaining to the public the limited character of the data. Instead, he and the Charter Commission have recklessly presented these results as evidence of widespread discontent in city hall and support for a change of government on the part of city employees. I finish with a quote from a letter to the editor Mr. Turner published in the Saratogian on September 10th.
“These results explain why 63.3 percent of City Hall employees said they believed city hall would operate better with a city manager.” No, Mr. Turner, they indicate that 48 out of 398 city employees ,roughly 12%, have indicated some level of support for a city manager form.