Start Menu

**The below study is strictly covered by NDA. Many details have been omitted or intentionally kept vague.**

Summary

PROBLEM

Prior to unveiling Windows 11, a product team from a large org wanted help deciding between two versions (v1 or v2) of the new centered start menu. Our team was asked to help them understand user sentiment and preference between these two versions.

METHOD

  • Survey
  • Concept test
  • Interview
  • Descriptive & inferential analysis
  • Thematic Analysis

RESEARCHERS

  • Lead UX Researcher
  • Anton Sirotin

MY RESPONSIBILITIES

  • Advised on/contributed to study design
  • Ran a pilot session as key stakeholders observed
  • Contributed to survey creation
  • Analyzed quantitative and qualitative data
  • Contributed to preliminary findings and insights
  • Contributed to final write-up and presentation

IMPACT

The strong signal we received from the survey made a direct impact on Microsoft’s decision to make the change from v1 to v2 despite strong business pressure in favor of v1. By making this change now rather than at the next Windows 11 update, we were able to effectively advocate for Windows users by delivering the products they wanted in a timely way.

Background

A product team within a core Windows org approached our team to help them understand user preference between the version of the new centered start menu in which they already made large investments (v1) and a second version (v2) that included adjustments based on hypotheses from the design team. This was a highly urgent project as the team came to us late in the product development process and the Windows 11 unveiling was months away. 

I was brought onto the project after the lead researcher had already scoped the study as well as built out much of the study design and survey itself. My role was to advise and contribute to study design, finalize the survey, run any pilot sessions, help analyze the data, and contribute to the write-up and presentation. While reviewing our study design, I noticed a slight discrepancy in one of the controls in one of our scenarios which lead to improvements in study design and more agreement from stakeholders.

We were aware of the pressures that stemmed from the large investments that were made into v1, but took seriously our role to advocate for the user by representing the data with scientific integrity.

Methods

SURVEY

The survey was conducted within Qualtrics and reached over 400 respondents. It consisted of different elements that evaluated start menu concepts across a randomized set of scenarios. Elements included (but were not limited to) concept tests; Likert scales to understand preference and degree of preference; and open-ended questions to understand user sentiment. 

CONCEPT TEST

The concept tests presented respondents with both start menu versions across several scenarios and solicited their feedback. Presentation of each concepts were followed by Likert scale questions which mapped onto Windows’ key design values. 

INTERVIEW

I ran a pilot session as key stakeholders observed. The session involved explaining the session to the participant before stepping back and observing them take the survey without my help or input. Concurrently, I moderated a discussion with the stakeholders in the chat. I encouraged them to ask any critical questions, which I combined with my own and posed to the participant at key times.

STATISTICAL ANALYSIS

Quantitative data were analyzed using both descriptive and inferential statistics using R and Excel. Descriptive analysis was done in R before the visualizations were created in Excel to conform to Microsoft’s design language. Before deciding whether to use parametric or non-parametric tests on our data, I check normality in R using the ggqqplot() function to create a Q-Q plot (quantile-quantile plot). Because the data were not normally distributed, I suggested we run a Wilcoxon to test significance between start menu versions within each scenario.

THEMATIC ANALYSIS

I analyzed all qualitative data using a thematic analysis to understand key themes within user sentiment across all open-ended responses. Themes were distilled and were used to contextualize preference measures.

Findings

We found there was a strong preference for the experimental version (v2) of the start menu across all scenarios. Further, all differences in preference between v1 and v2 were found to be significant.

Outcome

Despite the investments that were made in v1, senior leadership from the product teams used the insights from our study to successfully make a case that the start v1 should be updated before the Windows 11 announcement in June 2021. Today, the Windows 11 start menu looks the way it does in part