Waste of a Good Poll



Survey FBThe value of a poll question depends heavily on how you ask it. Take this “finding” from Forum Research, as reported by the Toronto Star:

“Ottawa’s tougher rules for…mortgages are a good idea, according to 63% of Canadians.”

That’s an interesting conclusion given the question that Forum Research asked:

“The Liberal government has tightened mortgage-lending regulations, and cancelled the primary residence tax exemption for foreign real estate buyers. Do you approve or disapprove of this decision?”

As written, the question combines two distinct rule changes: the new insured mortgage rules and the banning of the primary residence tax exemption for foreign real estate buyers. Combining the two has likely impacted the responses because a majority of people (rightly or wrongly) strongly support curbing foreign buyers. Even if some folks were not necessarily supportive of “tightened mortgage-lending regulations,” the grouped-in foreigner question could lead many to say they “approve.”

What’s more, the surveyor does not explain how “tightened mortgage-lending regulations” will reduce affordability, a big pain point for young buyers. Nor does Forum Research explain how such restrictions might jeopardize an existing homeowner’s equity. Nor does it touch on the potential economic ramifications of the rules, or even the future potential benefits for housing stability.

“We generally do not provide ‘context’ for fear of biasing the question and for fearing that we are providing information to respondents that the general population does not have,” says Forum Research President Lorne Bozinoff. But that cuts both ways. Had more people understood all of these points, and had the questions been asked separately, the “approval” numbers (for the mortgage changes specifically) might have looked rather different.

Knowing Canadians’ true feelings on one of the most impactful mortgage rules of all time would have been valuable. It’s too bad these pollsters dropped the ball.

Related Posts

Comments

  1. Comment avatar

    Ralph Cramdown    

    Reading your article, I concurred that conflating the two issues in one question may have been a poor choice.

    However, after looking at the survey results, I think your argument is moot. Higher education correlates strongly to more agreement with the policy change. Higher income correlates to more agreement with the policy change. In the face of that, it’s hard to argue that Canadians only agree with these policy changes because they’re poorly informed, don’t understand the possible consequences, or were unable to disentangle the two factors in the question. Per the foreigner angle, respondents who listed their ethnic background as “Canadian” agreed/disagreed/were unsure in the same proportions as all those who listed other backgrounds combined, and the HIGHEST “agree” numbers came from those who named their background as “Asian” (i.e. East Asian, as South Asian was a separate choice).

     
    1. Robert McLister    

      Higher income correlates to more agreement with the policy change.

      Which policy change? That’s the very question Ralph.

      Recall that an overwhelming 90% of Vancouverites supported the 15% foreign buyers tax: http://www.cbc.ca/news/canada/british-columbia/vancouverites-support-tax-poll-1.3700421

      The fact that a few specific demographics leaned one way on this commingled question implies little about the majority’s agreement with the insurance rules specifically. Given so many mortgage professionals are unsure about the potential outcomes, it’s hard to argue that most laypeople — no matter how rich/educated — have sufficient insight into how they’ll be affected.

       
      Comment avatar
      1. Comment avatar

        Reasonfirst    

        Wow – that is pretty condescending.

         
        1. Robert McLister    

          On the contrary RS, it’s likely quite factual. And note the word “most.”

           
          Comment avatar
      2. Comment avatar

        Ralph Cramdown    

        Virtually every demographic had a majority agree with the conjoined question, and “don’t know” was an option. The two exceptions were ethnic Black and income < $20k. The demographics I cited above had particularly high levels of agreement. You think that if the question was split, more would agree with the measures against foreign buyers and fewer with the other changes? You're probably right. But if you think it would be less than a majority, I'd say that's highly unlikely, given the numbers on the combined question. To say otherwise is to suggest that answers to both questions would have low correlation, or that Canadians' minds are so focused on the foreign buyer issue that it dominates their thinking about home prices in general. I don't think there's much basis for that.

         
        1. Robert McLister    

          Looks like we’re going in circles. 🙂 The takeaway was simple. A strong anti-foreign buyer sentiment has biased the responses to the second question. Despite our respective opinions, there is no way of knowing: A) the degree of bias (i.e., whether it would have been a minority or majority in agreement with all of the mortgage rule changes), and B) how the responses would differ if respondents were given more context on the changes.

           
          Comment avatar
        2. Ralph Cramdown    

          Well, we’re both trying to glean as much insight as we can from what we agree was a flawed survey.

           
          Comment avatar
  2. Comment avatar

    Ron Butler    

    I try never to comment on the “Ralph” posts. They are always the same boring one trick pony. This response was extra silly though. Anyone who does not understand the innate bias created by the precise wording of a survey question knows nothing about polling and surveys at all.

     
  3. Comment avatar

    Enumerator    

    Distinct concepts should always be sampled separately. Some surveyors combine questions simply to save money. Perhaps that was the case here. Each poll question can cost $1500 or more for 1000 responses depending on the polling company.

     

Leave a Reply

Your email address will not be published. Required fields are marked *

two + 1 =