Historically, pollsters have relied on samples drawn from lists of people with landlines. Many pollsters are now relying on a combined landline and cell phone list.
My initial thoughts on this approach are presented through a review of a PEW research study comparing landline and dual-frame election polls.
The Growing Gap between Landline and Dual Frame Election Polls:
Summary of Study: The PEW study looked at difference in poll outcomes for a poll based on a sample from a list of people with landlines and a sample based on a combined list of people with landlines and cell phones. PEW looks at the 2008 Presidential elections and the 2010 general Congressional results.
Main findings of the PEW study:
The 2008 election combined sample had the Obama-McCain margin at 8.2 percentage points. The landline only sample had this margin at 5.8 percentage points.
The 2010 poll results gave Republican candidates a 7.6 percentage point lead with the combined sample and a12.7 percentage point lead with the landline sample.
The 2010 analysis considers political preferences of four groups – landline only, duals from landline, duals from cell, and cell only. Groups from most to least Democratic are cell only, landline only, duals from cell, and duals from landline.
Comment One: The actual Obama versus McCain election differential in 2008 was 7.3 percentage points. This outcome is less than the margin predicted by the combined sample and more than the outcome predicted by the landline-only sample.
Comment Two: In my view, there is no such thing as a combined sample. We have two samples one drawn from the landline list and the other drawn from the cell list. This is not a semantic point.
Simply basing estimates on the combined sample means that the importance of cells versus landlines is entirely determined by the relative size of the two samples.
Since landline respondents tend to be Republican a landline-only poll is biased for Republicans. Since cell respondents tend to be Democratic the cell-only poll is biased for Democrats.
An unbiased poll would be a weighted average of the two polls with weights equal to the proportion of landline and cell phone voters.
Comment Three: The use of a combined sample leads to an incorrect (too small) estimate of the standard error. The standard error of the results should be based on the sum of variance of the two samples not a single combined sample.
(I’ll work out a simple math example on this point next week. Intuitively, the variance of the combine sample is the sum of the variance of all random binary individuals outcomes in the combined sample. The variance of the separate landline and cell phone surveys is the sum of the two variances. Sample size of both separate samples are larger so the variance of the two separate samples is larger than the variance of the combined sample.)
The use of the correct method of calculating the standard error for combine samples is especially important for analysis based on a single poll with likely voters only because the sample size for likely voters with cell phone only is quite small.
Comment Four: The 2010 results involve an aggregate of many elections. Hence it is difficult to compare 2010 landline/cell versus landline only results to actual outcomes.
Comment Five: The finding of difference between political preferences of duals reached by landline and duals reached by cell phone is difficult to evaluate without more information on why some people are being reached by landline and others are being reached by cell phone.
Comment Six: Duals from landline are by far the largest of the four groups. The next largest group duals from cell is 28% its size. These polls were still dominated by landline owners and users.
Final Comment: A lot of pollsters got it wrong in 2012, partially because they didn’t fully account for the growth of cell phone use. Many of these pollsters chose to move towards the combined landline/cell phone technique. This fix is inadequate.
The number of people dropping their cell phones continues to grow. I have some interesting data on this point that I plan to share either tomorrow or the next day depending on family obligations.