When Is A Survey Not A Survey: A Brief Tutorial Regarding Disinformation

There seems to still be confusion about the problems with the Charter Review Commission’s qua survey so in spite of my earlier post on the subject, I thought I would go into more detail here.

An entire industry has developed along with a huge academic discipline regarding how to accurately assess what a particular population believes about something by using surveys.

The classic debacle of the Truman vs Dewey election of 1948 is a sterling example of a survey gone wrong. In this case telephone interviews were done of a sample population to try to predict who the winner of the Presidential election would be that year. Newspapers printed headlines based on the results of that survey showing Dewey defeating Truman which obviously turned out to be wrong.  The flaw was that, at the time, telephones were something of a luxury.  By using that medium the sample was skewed to a group that was more affluent than the overall population and thus not representative of voters in general .

Wikipedia is usually a helpful resource for things like this and here is a link to their explanation about what the methodology for surveying is: https://en.wikipedia.org/wiki/Survey_methodology

When most people hear the word survey they naturally assume that it references the kind of survey explained in the above Wikipedia entry.

To begin with, there is the challenge of designing the questions.  Critical to this is having disinterested professionals craft questions so that they are free of language that would tend to prejudice the respondents.

Next is the selection of the sample population.  Usually the greater the sample, the more accurate the results.  Costs are commonly a limiting factor.  So a variety of strategies are developed to determine how to get a representative group.  In the case of city hall there would be a variety of problems in developing that sample.  For example, it would be reasonable to assume that certain positions would be more impacted by interactions with commissioners then others and therefore might have stronger opinions on the strengths or weaknesses of the current form of government.  The work of a clerk for example might make them less affected by decisions made by a commissioner than a mid management position.  There is also the factor of how well a particular employee might get along with a particular commissioner.  If they got along particularly well with a commissioner it might prejudice them to support the current form whereas if they did not they might be prejudiced to oppose it.  I am sure there are more issues but this is the kind of analysis that a rigorous professional in the field of survey methodology would be thinking about.  They would be struggling with how can they develop a model for sampling that gives an accurate picture of the population.

Another key element in a professional survey is to insure that the respondents are not allowed to be prejudiced in their responses.  There are a variety of tools and techniques applied to this process.  One of the things to control is that the respondents are reached in a manner and time that keeps them from speaking to one another about the survey.   The last thing a professional wants is for partisans with interests in the survey to affect the respondents.  As I understand it, some surveys where the risk of such interactions cannot be avoided build in a variety of algorithms to adjust the results accordingly.

It is axiomatic that the last thing a proper survey wants is self selecting respondents.  This is a reference to surveys where the participants are not selected according to the designed sample but where the sample is simply based on the people who want to fill out the form.  No adjustments are considered to deal with why some people decided to participate while others declined so it is impossible to determine to what extent those taking the initiative to answer are representative of the larger group.  They may in fact, like those with telephones in the Dewey example, share characteristics which set them aside from the population the survey is looking at.  Thus their responses will not be useful in determining what the group in general thinks.

To be blunt and obvious, no one would pay for a survey in an electoral race if they just sent out questionnaires  with a self addressed envelope to people in a particular district and asked them who they planned to vote for.  Even if they limited the mailing to registered voters or voters who cast ballots in the last two elections, what intelligent person would tabulate the results to determine whether a particular candidate was going to win based on who took the time to send the survey back?

So what did the Charter Review Commission do?

Did they seek the professional assistance from an expert in surveying to help them craft the questions?  No.  They crafted the questions themselves.

Did they consider that the questions might have hidden biases?  No.  They assumed that it was just a matter of asking some straight forward questions.  They were so sure of their good intentions they had no fear that there could be any problem with what they came up with.

So did they seek a professional to help them determine how many respondents they would need in order to draw conclusions about the target population (city employees)?   No.

Did they ask a professional how to determine what a representative sample of employees would constitute?  No.

Did they ask how they might solicit responses in such a way to insure that the survey would not be interfered with by partisans or that it could be done in a way in which respondents would not discuss the survey with others?  No.

What they did was to send out a questionnaire through the city’s computer network.  They didn’t even control that process.  When I asked Bob Turner, chair of the commission, who got the survey, he admitted he did not know.  Apparently there was some sort of internal city mailing list.  Pressed he was unable to offer anything about the nature of that mailing list nor did he know how many surveys were sent out. According to the Commission literature, it was a mailing list that went to employees in city hall.  Why did their survey not include all 398 employees who work for the city?  In some cases it was apparently  because some city employees do not have access to computers but this is not true of major departments that were overlooked. It does, though, expose one of the many limitations of using workplace computers to distribute the survey.

Still, the most damning thing about this survey was that its respondents were self selected.  This means that only employees motivated to respond did so.  According to the Charter Commission’s website seventy-five employees responded.  It is unclear how many employees work at city hall.  It is unclear what constituted working at city hall. Do the police work at city hall? Some do but others are out patrolling. How many? But this is the least of the problems with this survey.  The most serious problem is that it was self selected.

No person in good conscience could claim with certainty that the respondents to this survey were representative of the employees of city hall let alone the city.  The respondents of this survey were simply the seventy-five employees who chose for whatever reasons to answer the survey. No one has any idea how representative they may or may not be of city hall employees in general.

Sometimes the “Its Time Saratoga!” people flat out say that their survey found that 65.3% of the employees in city hall support having a city manager.  Other times they are more nuanced and just say that the survey was of city hall employees then give the numbers omitting that the numbers are a percentage of the 75 respondents only. (Since no one knows how many surveys were sent out we don’t even know how significant getting 75 responses was.)As should be abundantly clear, there is no way based on what was really simply a questionnaire that any conclusions can legitimately be drawn.

As just one of many examples, Bob Turner who chairs the Charter Commission published a letter in the September 10th Saratogian in which he asserted: “These results explain why 63.3percent [JK: The actual percent was 65.3] of City Hall employees said they believed city hall would operate better with a city manager.”  No caveats or qualifications here and patently untrue. 65.3% of city hall employees never told anyone anything. Turner can only assert truthfully that 65.3% of the respondents to that particular question answered it in that particular way. But 65.3% of 75 is only roughly 48 employees, not a number that the Charter supporters want to throw around.  Saying 12% of city employees answered a question saying they preferred a city manager form does not fit the pro charter change narrative.

Other times they simply say that their survey of city hall employees shows that 65.3% support a city manager.  I find this particularly cynical and disturbing.  The innocent reader would take it that the number represented a percentage of all city hall employees.  Instead, in a Clintonesque play of words, if challenged, they can claim they only meant the people who actually took the time to fill out their questionnaire and that they were using the word “survey” in the colloquial sense.  They didn’t mean a real survey.

Having hammered on this abuse of public trust repeatedly, their last mailing, buried among the large graphics and headlines about support for the city charter is the following: “The Commission’s survey and interviews with more than 75 employees found:”  Some might call that qualifying fragment progress.

Survey?  What survey?

8 thoughts on “When Is A Survey Not A Survey: A Brief Tutorial Regarding Disinformation”

  1. The Taliban would be proud of our Charter Commission. I don’t know if there is a legal recourse to their falsehoods, but it should be explored. Better yet, vote no and say that Saratogians are smarter than this. Make these guys 3-time losers like they deserve to be!


    1. I know which way I am voting tomorrow, it’s pretty obvious by what I have been reading–it’s not rocket science, kids….


  2. 2 thoughts:
    First I think this survey is emblematic of the campaign the charter commission and It’s Time have run. This phoney survey is not the only example of how they have misrepresented and cherry picked data. Whether it’s promising we will save hundreds of thousands of dollars on day one or blaming everything from Ayco’s departure to the unfinished bike trail on the commission form of government their literature,letters to the editor and statements to the press and on social media have in general reflected the same disregard for accuracy and integrity that is reflected in their use of this so-called survey.
    Secondly Bob Turner is the only one I would say knew for sure how flawed this survey is. After all sadly he teaches political science at Skidmore. So for me I see him as cynically using this flawed data to try to sell the charter proposal no matter what. But I’m wondering about all the others who repeated this false data. Did they know too that this survey showed absolutely nothing or did they just trust the professor that this data was sound and naively repeat it in letters to the editor and on social media ending up embarrassing themselves by repeating false information. In other words I guess I’m wondering if they were willing and knowing collaborators or if Bob Turner took advantage of their trust in him. We may never know.

    Sent from my iPad

    Liked by 1 person

    1. “In other words I guess I’m wondering if they were willing and knowing collaborators or if Bob Turner took advantage of their trust in him. We may never know.”

      Fact Checker: Bulls -eye!
      I would like you to join me in a pledge to keep looking in the hope that we MAY know. I take that pledge now. You too, host Kauffman!

      The fish rots from the head, some say in the movies. I say in real life – we have no place in the marketplace of ideas to promote falsehoods.

      I have not been shy about slamming Professor Turner’s leadership. But on one important level, it matters not whether the shame for this debacle should fall on him alone, or shame on some members of the Commission, or shame on all.

      For there is enough shame in this whole episode to go around.


      Which is why i am going out on a limb and hereby predicting that this proposition is going down.

      And surprisingly harder than expected perhaps. Yup.

      That ‘enough shame’ I referred to though, includes you – if you let this happen.

      If you decide to turn the keys of your government to this unelected gang, I am powerless to stop you. I am but a mere one-vote reporter. But I think Saratogians are smarter than to do that.

      With that faith in the electorate expressed – out on a limb, on election eve…

      Headline: Three-Time Loser

      You read it here first. See you at the polls.


  3. Do Skidmore students vote? That’s an interesting subject, all on its own. ‘Kinda’ like having the proverbial elephant in the room. Folks might do their best to ignore the creature but Jiminy Crickets! What a gift it leaves…indeed.


    1. I guess you don’t remember the days when Skidmore students numbers changed the outcome of an election. George Cannon went against Tom McTygue and Skidmore cost Cannon the election. Republican hierarchy tried for years to get the students from voting in Saratoga elections. You want them as part of the community, spend their money hear, eat out in our restaurants then you have to take it all.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: