When Is A Survey Not A Survey: A Brief Tutorial Regarding Disinformation

There seems to still be confusion about the problems with the Charter Review Commission’s qua survey so in spite of my earlier post on the subject, I thought I would go into more detail here.

An entire industry has developed along with a huge academic discipline regarding how to accurately assess what a particular population believes about something by using surveys.

The classic debacle of the Truman vs Dewey election of 1948 is a sterling example of a survey gone wrong. In this case telephone interviews were done of a sample population to try to predict who the winner of the Presidential election would be that year. Newspapers printed headlines based on the results of that survey showing Dewey defeating Truman which obviously turned out to be wrong.  The flaw was that, at the time, telephones were something of a luxury.  By using that medium the sample was skewed to a group that was more affluent than the overall population and thus not representative of voters in general .

Wikipedia is usually a helpful resource for things like this and here is a link to their explanation about what the methodology for surveying is: https://en.wikipedia.org/wiki/Survey_methodology

When most people hear the word survey they naturally assume that it references the kind of survey explained in the above Wikipedia entry.

To begin with, there is the challenge of designing the questions.  Critical to this is having disinterested professionals craft questions so that they are free of language that would tend to prejudice the respondents.

Next is the selection of the sample population.  Usually the greater the sample, the more accurate the results.  Costs are commonly a limiting factor.  So a variety of strategies are developed to determine how to get a representative group.  In the case of city hall there would be a variety of problems in developing that sample.  For example, it would be reasonable to assume that certain positions would be more impacted by interactions with commissioners then others and therefore might have stronger opinions on the strengths or weaknesses of the current form of government.  The work of a clerk for example might make them less affected by decisions made by a commissioner than a mid management position.  There is also the factor of how well a particular employee might get along with a particular commissioner.  If they got along particularly well with a commissioner it might prejudice them to support the current form whereas if they did not they might be prejudiced to oppose it.  I am sure there are more issues but this is the kind of analysis that a rigorous professional in the field of survey methodology would be thinking about.  They would be struggling with how can they develop a model for sampling that gives an accurate picture of the population.

Another key element in a professional survey is to insure that the respondents are not allowed to be prejudiced in their responses.  There are a variety of tools and techniques applied to this process.  One of the things to control is that the respondents are reached in a manner and time that keeps them from speaking to one another about the survey.   The last thing a professional wants is for partisans with interests in the survey to affect the respondents.  As I understand it, some surveys where the risk of such interactions cannot be avoided build in a variety of algorithms to adjust the results accordingly.

It is axiomatic that the last thing a proper survey wants is self selecting respondents.  This is a reference to surveys where the participants are not selected according to the designed sample but where the sample is simply based on the people who want to fill out the form.  No adjustments are considered to deal with why some people decided to participate while others declined so it is impossible to determine to what extent those taking the initiative to answer are representative of the larger group.  They may in fact, like those with telephones in the Dewey example, share characteristics which set them aside from the population the survey is looking at.  Thus their responses will not be useful in determining what the group in general thinks.

To be blunt and obvious, no one would pay for a survey in an electoral race if they just sent out questionnaires  with a self addressed envelope to people in a particular district and asked them who they planned to vote for.  Even if they limited the mailing to registered voters or voters who cast ballots in the last two elections, what intelligent person would tabulate the results to determine whether a particular candidate was going to win based on who took the time to send the survey back?

So what did the Charter Review Commission do?

Did they seek the professional assistance from an expert in surveying to help them craft the questions?  No.  They crafted the questions themselves.

Did they consider that the questions might have hidden biases?  No.  They assumed that it was just a matter of asking some straight forward questions.  They were so sure of their good intentions they had no fear that there could be any problem with what they came up with.

So did they seek a professional to help them determine how many respondents they would need in order to draw conclusions about the target population (city employees)?   No.

Did they ask a professional how to determine what a representative sample of employees would constitute?  No.

Did they ask how they might solicit responses in such a way to insure that the survey would not be interfered with by partisans or that it could be done in a way in which respondents would not discuss the survey with others?  No.

What they did was to send out a questionnaire through the city’s computer network.  They didn’t even control that process.  When I asked Bob Turner, chair of the commission, who got the survey, he admitted he did not know.  Apparently there was some sort of internal city mailing list.  Pressed he was unable to offer anything about the nature of that mailing list nor did he know how many surveys were sent out. According to the Commission literature, it was a mailing list that went to employees in city hall.  Why did their survey not include all 398 employees who work for the city?  In some cases it was apparently  because some city employees do not have access to computers but this is not true of major departments that were overlooked. It does, though, expose one of the many limitations of using workplace computers to distribute the survey.

Still, the most damning thing about this survey was that its respondents were self selected.  This means that only employees motivated to respond did so.  According to the Charter Commission’s website seventy-five employees responded.  It is unclear how many employees work at city hall.  It is unclear what constituted working at city hall. Do the police work at city hall? Some do but others are out patrolling. How many? But this is the least of the problems with this survey.  The most serious problem is that it was self selected.

No person in good conscience could claim with certainty that the respondents to this survey were representative of the employees of city hall let alone the city.  The respondents of this survey were simply the seventy-five employees who chose for whatever reasons to answer the survey. No one has any idea how representative they may or may not be of city hall employees in general.

Sometimes the “Its Time Saratoga!” people flat out say that their survey found that 65.3% of the employees in city hall support having a city manager.  Other times they are more nuanced and just say that the survey was of city hall employees then give the numbers omitting that the numbers are a percentage of the 75 respondents only. (Since no one knows how many surveys were sent out we don’t even know how significant getting 75 responses was.)As should be abundantly clear, there is no way based on what was really simply a questionnaire that any conclusions can legitimately be drawn.

As just one of many examples, Bob Turner who chairs the Charter Commission published a letter in the September 10^th Saratogian in which he asserted: “These results explain why 63.3percent [JK: The actual percent was 65.3] of City Hall employees said they believed city hall would operate better with a city manager.”  No caveats or qualifications here and patently untrue. 65.3% of city hall employees never told anyone anything. Turner can only assert truthfully that 65.3% of the respondents to that particular question answered it in that particular way. But 65.3% of 75 is only roughly 48 employees, not a number that the Charter supporters want to throw around.  Saying 12% of city employees answered a question saying they preferred a city manager form does not fit the pro charter change narrative.

Other times they simply say that their survey of city hall employees shows that 65.3% support a city manager.  I find this particularly cynical and disturbing.  The innocent reader would take it that the number represented a percentage of all city hall employees.  Instead, in a Clintonesque play of words, if challenged, they can claim they only meant the people who actually took the time to fill out their questionnaire and that they were using the word “survey” in the colloquial sense.  They didn’t mean a real survey.

Having hammered on this abuse of public trust repeatedly, their last mailing, buried among the large graphics and headlines about support for the city charter is the following: “The Commission’s survey and interviews with more than 75 employees found:”  Some might call that qualifying fragment progress.

Survey?  What survey?