Advanced Quant Techniques with Small Samples: The Right Way
Ben Cortese, our VP of Decision Sciences and Innovation, brings more than a decade of experience in advanced quantitative methodologies and a passion for uncovering hidden insights through data to KS&R. Ben leads a close-knit team of expert data scientists and analysts that are the backbone of quantitative research for the organization, providing full-service consultation and analytics to our clients. But what truly sets him apart is his quantitative thought leadership, tackling challenging problems such as pricing research, feature prioritization, and predictive analytics through novel and innovative techniques. While his Ph.D. in statistics from Syracuse University is a testament to his dedication to the craft, Ben is about more than just numbers. A proud father and husband, he cherishes time outdoors with his family, spending time in the garden, and playing golf during Syracuse's fleeting sunny spells. For Ben, it's not just about data – it's about balance and the joy of life's simple pleasures.
Allison is an experienced director with a penchant for unraveling complex datasets and distilling them into actionable insights. With a background in Cognitive Neuroscience and Statistics, she's KS&R’s go-to quantitative expert, especially within the Healthcare Industry Team. Allie's proficiency in machine learning and diverse statistical techniques makes her an asset across all industry verticals. She excels in methodologies such as segmentation, conjoint analysis, and advanced modeling. Outside of work, Allie enjoys outdoor pursuits with her husband and two English bulldogs. Exploring nature and discovering dog-friendly city spots are her passions. Allie's blend of analytical expertise and genuine enthusiasm enriches both our team and the solutions we deliver to clients.
In the market research industry, there is a lack of consensus when it comes to sample size guidelines and guardrails, especially when conducting advanced quantitative research (think anything beyond stat testing). A common question arises: “How small is too small?” to achieve a statistically significant or worthwhile result. To tackle this challenge, we set out on a journey to define clear sample size recommendations for traditional online research and explore the potential benefits of mixed mode research – collecting data for advanced quant techniques through qualitative interviews.
The Micro Modeling Approach
This initiative led to the development of “Micro Modeling”, a framework for executing advanced quantitative research with small samples. Micro Modeling provides guidance by addressing several key questions:
- What sample size is necessary to have confidence in MaxDiff and Conjoint results?
- Do respondents provide higher quality quantitative data when collected via an In Depth Interview (IDI)?
- Can we produce valid results through mixed mode research with smaller samples than traditional online research?
- Are there limitations or guardrails to keep in mind when executing this type of research?
Healthcare Case Study
Efforts were focused within the healthcare space, as HCP data is difficult and costly to obtain, especially when conducting research in rare disease states. There are broad applications beyond healthcare in tech, B2B, and any other industry where the population is limited or sample is scarce. Our rigorous, two-phased approach consisted of a large-scale quantitative study using an online survey, alongside a qualitative phase, with over 500 total cardiologists interviewed. Data collection was facilitated by our partners at EMI, ensuring robust and diversified inputs.
Results and Recommendations
Results from the quantitative phase were clear. Recommendations for MaxDiff fall on a spectrum where smaller base sizes (even smaller than we expected ) do indeed provide valuable insights, even if somewhat limited. As expected, larger base sizes drive more confidence. A straightforward summary of results demonstrates some key sample size thresholds.
Unfortunately, we had limited success with conjoint and were unable to validate results with smaller samples than historic benchmarks. Therefore, we suggest maintaining the status quo when conducting conjoint research through online surveys.
The second phase focused on further reducing recommended thresholds when data is collected through qualitative interviews. Similar to the quantitative phase, there is evidence of an opportunity to leverage small samples, as we observed reliable outcomes for MaxDiff with small base sizes using a mixed mode approach. The recommended thresholds achieved through Micro Modeling are in fact smaller, a strong indicator that conducting a tradeoff activity via an IDI improves the quality of data.
The similarities between the quant and qual phase extend to conjoint as well. We were unable to validate a viable small sample solution, even with data collected via an IDI.
Additional Benefits
Advanced quantitative analysis with small samples is commonplace, and with Micro Modeling, we have established a framework and guidelines, validated via rigorous statistical analysis, to execute MaxDiff. Added confidence in this methodology with small samples was our key objective, but there are several additional benefits extending the utility of Micro Modeling.
- Quant Insights from Small Samples: Qualitative studies with as few as 15 IDIs can yield at least some level of viable quantitative insights.
- Extending Qualitative Research: Micro Modeling bolsters qualitative insights through a quantitative lens which is especially useful for message testing and feature prioritization.
- Enhancing the “Why”: Through Micro Modeling, participants explain their thinking, providing an in depth understanding of the decisions made in a prioritization activity.
Micro Modeling’s 5 Big Things
Micro Modeling was successful at defining a framework for executing a MaxDiff activity with small samples. When considering Micro Modeling, keep these five big things in mind.
- Validated. Our results indicate that Micro Modeling via MaxDiff is viable.
- Better Quality Data with Qual. Using data collected through IDIs, MaxDiff models performed better than those run with data collected through online surveys.
- Confidence is Tied to Base Size. We suggest five distinct thresholds balancing precision and sample size for MaxDiff studies.
- Big Quant Studies Still Have a Place. Large quantitative studies provide a level of accuracy and precision that is not supported by Micro Modeling.
- Complex Models Need Bigger Base Sizes. Analysis of conjoint models requires larger base sizes than Micro Modeling, despite higher quality data.
