Addressing the "Landline Only" Polling Myth

This is the first in a series of articles examining polls and other issues related to the Democratic presidential primaries (originally published here).



Sep 2019

“Polls are done using landlines. I can’t even think of 5 people I know who have landlines,” wrote a disgruntled podcast viewer. “If they poll on mobiles or the internet and not landlines, [candidate] would win hands down,” bemoaned another.

Comments like these appeared regularly in my social media feeds the past few weeks. They suggested that Democratic primary polls returned biased results because they exclusively sampled landline users, whose demographics don’t mirror the general electorate. As someone with limited background in polling, my initial reaction to this was shock. Was our polling infrastructure fundamentally flawed? It seemed bizarre that landlines were still used to reach poll participants when cell phones have all but replaced them. So I decided to study the data for myself to see if these claims were actually based in fact.

I set out to analyze FiveThirtyEight’s collection of 400+ primary polls from the past year, closely inspecting their results and methodologies. It didn’t take long to realize that poll participants were actually reached a variety of ways. These included calling cell and landline numbers as well as contact via the internet, text, and traditional mail. So as far as the “landline only” controversy was concerned, that myth was quickly debunked. However, eager to better understand the polling process, I continued exploring the data. The charts below present a summary of my findings.

This first chart provides a breakdown of how individual pollsters reached out to participants:

Note: Unknown breakdowns are classified as 50%-50% (e.g. Siena College) or split into thirds (e.g. Gravis).

As this visual shows, landline calls (represented by orange bars) are a popular polling method but constitute just one part of the overall landscape. Cell phone calls and internet polls are also heavily represented.

The next chart presents a more macro view, capturing how many pollsters used a particular method:



Phone polls (cell & landline) remain the most popular approach overall, with online polls not far behind. However, since online polls are generally cheaper and quicker to produce, a majority of active pollsters rely on them, as is evident in the chart below.



Concluding Remarks:

It should be noted that even a single method for reaching participants can be implemented in a variety of ways. For example, a few pollsters use Interactive Voice Response (IVR) to reach phone poll participants, which may produce different results than traditional human phone polling. Furthermore, it’s important to emphasize that polling is a complex process comprised of many layers beyond what was presented above. These include factors like how questions are worded, participant response rates, and how sample weighting is conducted, among others. A meaningful examination of poll results should also consider these elements. With that in mind, the analysis presented here should be viewed as an introduction to one aspect of primary polling, which hopefully cleared up some misconceptions about how they are conducted.

In part 2, I present a few additional observations, specifically shedding light on DNC-approved polls.

Further Reading:

*Any comments, corrections, suggestions, etc. are always welcome. The code and details behind my analysis can be found here.