The major news outlets, not surprisingly, have missed the point on the upcoming statistical survey of homelessness in Toronto. The headline-grabbing story tells about $100 gift cards being given to 100 people who will act as homeless decoys during the night of the April 15 survey in order to provide a measure of statistical validity to the study. This sort of introduction of "defects" in quality control style studies is the source of that famous disclaimer, "accurate to within +/- 3%, 19 times out of 20." The real story is that the survey methodology - as statistically valid as it might be - is relatively worthless when it comes to understanding the real needs of the homeless population.
The reason has to do with the limitations of positivist methodologies (and when you're talking about statistical significance, you're talking about positivism) in general, and the specific biases introduced when researching marginalized populations. Simply put, when you are dealing with people, the researcher has no way to know whether answers to survey questions are truthful. This problem is exacerbated when dealing with vulnerable or marginalized populations - those who have reason to fear or mistrust. When doing an ethnographic study (which is essentially what this homeless needs assessment is), it is important to establish a relationship of trust between the researcher and the participant. Otherwise, the participant may simply say what s/he thinks the researcher wants to hear. In this case, some of the people being surveyed may answer, or not, out of fear, or make up answers as an evening's entertainment. The statistical analysis can tell you whether the numbers work, or not; it can't tell whether the answers are reliable, valid, truthful, and worthy of forming the foundation of public policy. And the stock response of, "it's better than what we have now" is a nonsensical justification for ill-conceived policy that does little to address the complexity of the sources of homelessness in this city.
"Councillor Janet Davis (Beaches East York) said the methodology was designed by academics who are experts on statistics." Perhaps, Councillor Davis, you should be using a methodology designed by academics who are experts on marginalized populations to understand the needs of marginalized populations.
Update (2 Apr 2009): Iain de Jong, Manager of Toronto's Streets to Homes program responds.
[Technorati tags: toronto | city council | homeless | survey | janet davis]
7 comments:
Straw men and righteous indignation go together quite well with the anti-intellectual, common sense, plain talking folk crowd.
This is a case in point; the issue isn’t in spending $5000 to have people pretend they’re homeless (5% of the 100K budget for a homeless survey). The issue should be about the efficacy of doing this survey in the first place. I don’t have enough information to decide that, but hey! not my city.
Commenting on and casting judgement on whether it should be done in the first place is fair game.
Casting derision on a statistical research method should be left to those who understand how numbers work.
Putting “decoys” out in the field and assessing how many of them get approached by fieldworkers increases the validity and significance of the survey. It. Simply. Does. I could point you to a page that talks about experimental design, but really, I doubt many would understand it. It was hard for me when I studied it (and used it in my own research) back in the day.
Which is the real point here. On the surface, this seems like an easy outrage to attack. “How dare they pay someone to pretend to be homeless when there are real homeless people out there!” they decry. The irony of this all is that the very same whiny voices one hears, outraged by this seeming misuse of money, are the very same who will normally complain about the homeless in the first place. Talk about arguing both side of the equation. A very thoughtful approach, that.
If you are uneducated in science, or mathematics, or experimental design, you really need to chill a bit and let those who do know a thing or two about those arcane issues do their job.
No really, if you gave up on numbers or formulii or other complicated subjects in high school or university, you have lost your right to comment intelligently.
Notice I’m not commenting on spending $100K on surveying the homeless. That’s an entirely separate issue. I can see both pros and cons to the approach, and am willing to let the experts in that field make their case.
You seem to know a bit about what you comment about, which is refreshing. The howling masses, though, really piss me off!
Wow, Alexi. For someone who sounds vaguely intelligent, I'm surprised at the degree of your illiteracy (i.e., can't comprehend my post), your lack of understanding context, and your very limited understanding of research paradigms (i.e., understanding only positivist methods, and not constructivist, critical or radical methods).
As for my own résumé, no I am not uneducated in science, math or experimental design. Quite the opposite, in fact. I graduated Engineering Science, so I am quite versed in science and advanced mathematics, including several courses in statistics. I spent a good decade early in my career doing econometric modelling and capacity/load forecasting. I have also had the benefit of learning research design among a variety of both quantitative and qualitative methods, across positivist, post-positivist, constructivist, critical, and radical paradigms. I am now completing a PhD at the University of Toronto, and guess what - my study is an empirical study (that means, designed research where I go out and, you know, study people).
I claim some small expertise in the area of methodological paradigms, perhaps evinced by the fact that I have been invited to give the keynote address at several medical research conferences in Western Canada on the limitations of the scientific method and statistical analyses when dealing with sociological issues, and those that are better understood using the principles of complexity - and the complex problem of homelessness in Toronto certainly falls under both categories. So yes, Alexi, I have the right to comment intelligently.
Man, do I hate having to trot out credentialism, especially for someone who seems to be intelligent but is, in actuality, elitist and functionally illiterate.
Back to my point: I am not arguing the statistical validity of the study. I'm sure the academic statisticians whom Councillor Davis employed know their craft. What I question is the validity of a positivist (or post-positivist) approach to the problem of researching a marginalized community. Other paradigms and methodologies better lend themselves to the nature of the problem. It's a simple matter of instrumentation: one can only discover that for which one's instrument is calibrated. Statistics is very useful for many things. However, it is not a universal instrument of discovery.
Dude. Chill. Read my last para. I agree with you (mostly). If I'm bleeting, it's about the others who are attacking the wrong issue.
And yes. Yours was the only blog posting on the issue that I've read that deals with the real issue at hand. I should have made that more clear, but I presume that people read the entirety of what I write.
We good now?
Like most arguments, you are both right. However, despite Iain de Jong's poor defense of it, the Street Count also has some merit: it gives heft to the political fight for the marginalized. "Hard numbers," however earnestly derived, are an important part of the means of getting attention and resources to an issue.
Thanks all for your comments.
(Now, I don't have to trot out my credentials too, do I?)
I agree, Diane, that there is merit in numbers. But like any empirical approach, numbers only provide one aspect and tell only a very small part of the overall story. The problem is that those who are not well-versed in research, the history and philosophy of science, and the limitations of abstract empiricism (i.e., politicians and policy makers), tend to believe that numbers are the ne plus ultra of providing understanding; that stance is just plain wrong. de Jong does a terrible disservice to the entire citizenry, not just the marginalized, through his stance of defensiveness, rather than promoting a multi-methodological approach to a complex problem.
Or, put another way, you cannot solve a complex problem with complicated tools.
I agree! I cannot stand De Jong! I do not agree with what he says. He is offensive and uses this capability to attract an audience . I had the unpleasant honour of attending a conference where he was speaking. He is loud and his theatrics were unsavy and rude. A doof, a dolt, and a sure ass idiot!
Post a Comment