A high-stakes survey study that directly influenced Microsoft's decision to change the Windows 11 Start Menu before its public announcement — overcoming business pressure in favor of the original design.
A product team within a core Windows org approached our team late in the product development cycle — the Windows 11 announcement was only months away. They had already made large investments in v1 of the new centered Start Menu, but the design team had hypotheses that v2 would perform better with users. They needed data, fast.
I was brought onto the project after the lead researcher had scoped the study and built much of the survey. My role was to advise on study design, finalize the survey, run pilot sessions, analyze data, and contribute to the write-up.
We were acutely aware of the business pressure surrounding v1's investments, but remained committed to representing the data with scientific integrity.
The study design included testing both start menu versions across several conditions (i.e., desktop scenarios). While reviewing the conditions, I identified a discrepancy in one of the scenario controls — flagging it led to improvements in study rigor and greater stakeholder confidence in the results.
Before selecting statistical tests, I checked normality in R using ggqqplot() to generate Q-Q plots. Because the data were not normally distributed, I proposed using a Wilcoxon signed-rank test to evaluate significance between start menu versions within each scenario — a methodological choice that strengthened the credibility of our findings.
There was a strong and consistent preference for v2 across all tested scenarios. Critically, the differences in preference between v1 and v2 were statistically significant across every scenario — not just directionally favorable.
Thematic analysis of open-ended responses provided qualitative context for the quantitative signal, helping the team understand not just what users preferred but why — and how to communicate that to senior leadership.
Despite substantial investments in v1, senior leadership used our findings to make the case internally for switching to v2 before the Windows 11 announcement in June 2021.
The Windows 11 Start Menu looks the way it does today in part because of this study. By acting on the data early, the team avoided shipping a suboptimal product — and saved significant post-launch engineering cost.