The major news outlets, not surprisingly, have missed the point on the upcoming statistical survey of homelessness in Toronto. The headline-grabbing story tells about $100 gift cards being given to 100 people who will act as homeless decoys during the night of the April 15 survey in order to provide a measure of statistical validity to the study. This sort of introduction of "defects" in quality control style studies is the source of that famous disclaimer, "accurate to within +/- 3%, 19 times out of 20." The real story is that the survey methodology - as statistically valid as it might be - is relatively worthless when it comes to understanding the real needs of the homeless population.
The reason has to do with the limitations of positivist methodologies (and when you're talking about statistical significance, you're talking about positivism) in general, and the specific biases introduced when researching marginalized populations. Simply put, when you are dealing with people, the researcher has no way to know whether answers to survey questions are truthful. This problem is exacerbated when dealing with vulnerable or marginalized populations - those who have reason to fear or mistrust. When doing an ethnographic study (which is essentially what this homeless needs assessment is), it is important to establish a relationship of trust between the researcher and the participant. Otherwise, the participant may simply say what s/he thinks the researcher wants to hear. In this case, some of the people being surveyed may answer, or not, out of fear, or make up answers as an evening's entertainment. The statistical analysis can tell you whether the numbers work, or not; it can't tell whether the answers are reliable, valid, truthful, and worthy of forming the foundation of public policy. And the stock response of, "it's better than what we have now" is a nonsensical justification for ill-conceived policy that does little to address the complexity of the sources of homelessness in this city.
"Councillor Janet Davis (Beaches East York) said the methodology was designed by academics who are experts on statistics." Perhaps, Councillor Davis, you should be using a methodology designed by academics who are experts on marginalized populations to understand the needs of marginalized populations.
Update (2 Apr 2009): Iain de Jong, Manager of Toronto's Streets to Homes program responds.
[Technorati tags: toronto | city council | homeless | survey | janet davis]