The following are notes that I took live while Frank Graves (Owner and President, EKOS Research) and Anne Crassweller (President, NADbank) presented at MRIA Netgain 8.0. There might be typos in the posting.
The post also includes video clips of Anne Crassweller and Frank Graves, providing overviews of their presentation
Aiming for Gold
Probability still matters: Using HD-IVR to Reach all Canadians
Traditional MR seems almost old-fashioned these days, but “counting counts”.
- MR industry needs to aim for gold.
- Needs to provide market research that provides insights for clients not just data.
Media uses data for their daily advertising, so how population uses newspapers, radio etc is important. Without representative research cannot predict how many people read the newspaper.
Do not want to do research like in the past, since consumer has changed. Need to use new tools available to be able to do so.
- Almost 20% of Canadian households are cell phones only, problem for CATI.
- Long lag times between collection of data and delivery.
- Response rates declining.
- Cost of conducting research need driven up.
- Stakeholders want data overnight.
Consumers have adopted the internet, but has not created a statistically representative sample.
EKOS has created a new RDD method for recruiting respondents.
Looking at a test to see how well the technology worked for the research problem.
Frank Graves, Owner and President, EKOS Research, Ottawa
AAPOR consensus is that opt-in online panels:
- are not reflective of households
- yield low and unknown response rates
- are subject to coverage, non-response and selection bias that cannot be corrected for
- should not report margin of error as it is misleading
High Definition (HD) Interactive Voice Response (IVR)
- dual frame sampling (landline and cell only)
- rigorous randomization of response categories
- experimentally optimized call-backs
- last birthday + household size to represent individuals, not households
- method for detecting and removing noise/erratic responses
- experimental innovations, e.g. incentives
Checking accuracy of results for polling is not necessarily a good yardstick, since voting turnout in some cases is as low as 50%.
- Including offline respondents can get a better convergence with reality, because some people are systematically different than others.
- HD-IVR has provided some results very similar to census data, often closer than CATI on age, gender and education figures compared to census.
- Comparison between NADbank and HD-IVR on Newspaper Readership
- Print numbers for the three of the four major Toronto dailies were similar between the NADbank 2012 study and HD-IVR data.
- More work to be done to ensure that random opt-in probability sampling is used.