Survey Results From Task Force on Police Reform: Flawed Survey Produces Worse Findings

The Times Union published a news story written by Wendy Liberatore on March 12, regarding the findings of a survey done by John Schroeder on behalf of the Saratoga Springs Police Reform Task Force.

In an earlier post I reported on the glaring problems with the survey. Regrettably, Ms. Liberatore was apparently uninterested in critically examining the survey and wrote about it as though the numbers presented to her by Mr. Schroeder were an accurate representation of the opinions of the citizens of Saratoga Springs.

The story extensively interviewed Mr. Schroeder. The article highlighted statistics from the survey that purported to support allegations regarding racism and excessive force by the Saratoga Springs Police Department. It included very disturbing narratives offered by persons who participated in the surveys. It finished with the following quote from Mr. Schroeder:

“Data drives out opinions,” Schroeder said. “City councils and politicians tend to listen to whoever shouts last loudest. … But they can’t argue with the data.”

Times Union March 12, 2021

With respect to Mr. Schroeder, data can drive out opinion but only if the data is accurate and reliable.

The survey results are available here.

Some Background Thoughts On The Task Force

I took the time to watch the February 24, 2021, meeting of the Task Force. This is a link to the video. I came away from the experience impressed by the performances of its co-chairs Jason Golub and Camille Daniels and the committee. The Task Force is quite diverse but it was clear that respect for each other was the order of the day. The discussions were substantive. It was apparent that the members of the task force were dedicated to both social justice and to developing policies that they believe would enhance the work of the city’s police department.

The work of the Task Force has been particularly challenging in that they had no funding for professional support and were attempting to take on some very difficult issues within a very limited time period.

With that in mind I would urge the readers of the blog to distinguish between the work of the Task Force and the problematic character of the survey.

Mr. Schroeder is not a member of the Task Force and developed the survey as a volunteer. As far as I can tell, he has no expertise in polling. During the last few years Mr. Schroeder has been an aggressive critic of the Saratoga Springs Police Force. He has extensively FOILed the city for police department documents and recently won a suit against the city for failing to provide some documents within the time restraints of the Open Meetings Law.

The “Survey”: Beyond Flawed

There is a science to polling (flawed as that science may be). Social scientists who do polling direct great resources to selecting participants who will be representative of the group they seek to understand.

Mr. Schroeder published his analysis of the survey results with the following caveat about the participants in his survey:

While selection bias is always a concern for open surveys, the demographics are very close to our overall population. 

Introduction To Survey Report

The results that follow are statistically significant and broadly representative of the overall City of Saratoga Springs Community. 

Introduction To Survey

Mr. Schroeder is referring to the fact that the breakdown of gender and some other characteristics of who took the survey were similar to the demographics of the city.

Mr. Schroeder asserts that because some of the self identified characteristics of those who took the survey are similar to the demographics of the city, that this somehow establishes that the survey results are a valid reflection of what the residents of Saratoga Springs think. Aside from the fact that the only characteristic that even is close to correlating with the city’s population is the number of persons who self identified as being White, without proper modeling, any similarity between the self identified characteristics (age, gender, race, etc. ) of the participants of his survey and the actual population of our city is simply coincidental.

It is disturbing that he is trying to reassure us that the survey is “statistically significant ” and that it is “representative of the overall City of Saratoga Springs” simply because those who took the survey in some cases resemble the population of the city.

It is particularly problematic that he thinks his survey’s results indicate what the city of Saratoga Springs thinks of its police department given that a significant number of people who took the survey do not even live in Saratoga Springs.

The survey asked the participants where they lived. Here is one of the tables:

So only 57.9% of those taking the survey identified themselves as actual residents of Saratoga Springs. The notes at the bottom of the graph assert that the people who live outside of the city but live in Saratoga County are “stakeholders” so, the argument goes, they should be included. Maybe, and some people may find this nit-picking, but Mr. Schroeder asserts that the survey results are “representative of the overall City of Saratoga Springs” when roughly forty percent of the participants were self described as living outside the city. This kind of sloppy, indifference to his own statistics is antithetical to the kind of rigor required in proper polling.

Participants in Mr. Schroeder’s survey were “self selected” which is to say that the participants were whoever was motivated to take the poll. It was not a sampling that necessarily represented the city’s population.

Also troubling was that there were no controls to keep participants from taking the survey more than once. I know that after doing the survey once I was able to log on to do it again if I had wanted to without a problem. I did not repeat the survey.

This is no minor technical problem. It strikes at the very integrity of the results.

With respect to Mr. Schroeder, I cannot understand how he can can describe a self-selecting survey (no modeling or randomization) as “representative of of the overall City of Saratoga Springs.”

In addition to the unscientific sampling techniques used, individual questions in the survey also violate the fundamentals of sound polling. I explored that in some detail in an earlier post but I will revisit one example.

The survey asked the participants:

Please indicate which of the following changes to policing you support:

Participants in the survey were limited to the following responses:

Support – Don’t Support – Not Sure

One of the statements asks whether the person being surveyed supports/doesn’t support/is not sure about Ending profiling, “Stop and Frisk”, and policing of minor issues

First of all, a proper polling question does not include three items as is done here. What if the respondent supports or doesn’t support or is not sure of only one or two of the three items? How can a response like that be registered?

The question also implies that the city currently employs all three of these items by asking about “changes to police policy” . This is similar to the classic question, “when did you stop beating your wife?” Where is Mr. Schroeder’s data supporting the allegation of profiling or Stop and Search in the city? And what does “policing of minor issues” mean?

As the question asks simply about police rather than the Saratoga Springs police, the question invites confusion. When I answered this question I assumed I was being asked about our local police. This is another example of how poorly this survey was crafted.

If you want to improve service delivery, or anything else for that matter, one of the first steps is to understand where you are today. Without that baseline,it’s hard to tell if things are improving or getting worse over time. This survey is intended to provide that quantitative understanding of how our entire community views the Saratoga Springs Police Department (SSPD). Our hope is that this becomes an annual exercise that can chart the progress we make together.

Introduction To Survey

Unfortunately without a scientific sample and carefully crafted questions, there is no baseline.

While the previous “up and to the right chart” showing overall satisfaction is visually appealing, it is equally true that nearly 1 in 4 people in our community are dissatisfied with the SSPD

Introduction To Survey

The assertion that nearly “… 1 in 4 of the people in our community are dissatisfied with the SSPD” is simply not an established fact. It is only accurate to say that 1 in 4 of the people who took this survey are dissatisfied.

Trying To Unspin The Spin

Question #8 in the survey asks:

“Have you or someone you know experienced physical mistreatment, harassment or intimidation by a police officer?”

So this is a strangely global question for a survey trying to get information about the Saratoga Springs Police Department. There is no limit as to when (1969? last month?)or where (Guatemala? California?) an incident could have occurred. It isn’t even simply about whether the person taking the survey has had a bad experience with the police anywhere at any time. It could be the person answering or anyone one they have ever known. I have to admit, given how broad the question is, I am surprised that only a third of those surveyed said yes.

So Mr. Schroeder then correlated the number of people who answered yes to the police experience question with how satisfied those persons were with the Saratoga Springs Police Department. According to the two charts above, if you answered Yes to the bad police experience question you were much more likely to be dissatisfied with the Saratoga Springs Police Department. If you answered Yes and you “considered yourself a member of an historically excluded group (e.g. person of color, LBGTQ)” the rate of dissatisfaction increased.

So what does this mean? As far as I can tell it means that if you or someone you know had a bad experience with the police somewhere and at sometime, then you will tend to look unfavorably on the Saratoga Springs Police Department. How is this correlation meaningful?

At the risk of appearing cynical, I think Mr. Schroeder wanted to have another chart that presented an image of people unhappy with our local police in order to compensate for the responses that pretty uniformly show that overall people of all races who took the survey had a favorable view of the local police (see chart below). The rate of satisfaction for people who self identified as “Black/African American” was pretty much the same as those who self-described as “White.”

Some Final Thoughts

The analysis had two appendices. One involved comments from people elaborating on their, or someone they know’s, bad experiences with the police in general (any police not just the local police). The other appendix involved suggestions on how to improve our local police department.

In both cases the appendices had literally hundreds of entries. Granted the allegations against the police are anonymous, but however true or accurate they are, they are a reminder of America’s troubling history of racism.

I find these appendices the most interesting and the most encouraging elements of the survey.

It makes one really proud to be a member of the Saratoga Springs community that so many people are civically involved and care so deeply about both our city and the need to bring social justice to our country that they not only took the survey but had the initiative to offer their insights to the task force.

---

This is the first of seventeen pages in the appendix on police abuse.