← Return to Liz's Work

Research Case Study

Ensure Successful Product Launch through a Beta Process

Background

The product involved in this case study was launched by IT professionals to the end-users of their systems. Its purpose is for end-users to authenticate into their work systems so that they stayed secure from breaches and bad actors. 

Internally, we had a robust persona definition used across Research, Design, Product and Engineering. Our end-user persona was called Lee and our IT professional was called Gary. 

I had led previous research that revealed the prompt needed an update that would allow for an improved user experience which included accessibility improvements, internationalization to accommodate more languages, and visual customization options.

The organization would conduct a beta program to release the initial design that had been crafted through upfront research and concept testing. While it was in beta, my role was to measure the efficiency and effectiveness of the new authentication experience from the point of view of IT professionals who would be launching this product, as well as the internal end-users who would be authenticating through it at least once a day.

For the product to be considered a success, we needed both Gary (IT professional) and Lee (end-user) to consider the product easy and effective.

Process

Method

I built a mixed-methods plan that would measure end-user attitudes and behaviors through interviews, surveys, analytics and data coming from support tickets. The plan also included interviews with IT professionals to ensure that they found it easy to implement and manage their end-users interactions with the tools. 

We would use this feedback to inform the next round of design iteration for the authentication experience. 

As I built out the research plan, I wanted to ensure we had clear expectations of what metrics would define success.

Defining Metrics  

At the time, product managers did not use a standard framework to measure success. I was tasked with finding a measurement framework that would help us define and measure success. 

I chose the HEART framework because:

  • It enables definition of goals that allow you to evaluate impact in terms of business objectives and user experience

  • It’s customizable to the needs of the project

    • Teams can choose the categories that make sense for their needs, and define Goals, Signals, and Metrics 

  • It allows for triangulation among qualitative & quantitative metrics 

Because this was the first time that the team was going through the process of measuring analytics for success, I knew that I would need to involve the team to get buy-in.

I designed and conducted a workshop (you can read more about it here) and we defined the following metrics in the categories of Happiness and Engagement. 

Happiness
Goal: Lee (end-user) feels authenticating with Prompt 4 is easy 

  • Signal: Vast majority of Lees rate Prompt 4 as easier than Prompt 3 on survey

      1. Metric: statistically averaged ease of use score for Prompt 4 is higher than for Prompt 3
        with relatively small standard deviation.

    • Signal: Minimal Prompt 4 support requests

      1. Metric: Customers will report how many support requests they receive related to Prompt 4

      2. Metric: Customers will report the severity and nature of support requests received related to Prompt 4 

Engagement
Goal: Lee spends less time authenticating with Duo 

  • Signal: Overall time spent in Duo Prompt reduced 

    • Metric: track time from loading prompt to exiting the prompt.

    • Metric: average time per Lee spent viewing Prompt 4 is much lower than Prompt 3 

Outcome

Specifically related to the research questions and metrics, we found that the attitude and behavioral measurements showed the new authentication prompt was perceived as easier and faster, and analytics showed that users were moving through it faster.

Qualitative interviews with Lee (end-user) and Gary (IT) revealed additional insights and considerations that we needed to take into account for our customers to successfully transition to this product at scale. Some highlights include:

  • We needed Gary to be motivated to adopt our new authentication process in a timely manner in order to avoid a costly stage of transition. Clarity of timeline, process, features and benefits would be critical. Doing it well would require a lot of upfront work, thoughtful preparation, and alignment on implementation across our disciplines. 

  • We needed to help Gary mitigate risk. With IT rollouts, risk mitigation is paramount, and Gary needed the ability to preview the experience before it went live and see how it would affect his end-users.

  • During key interactions, we needed to clearly set expectations for Lee’s next steps and give cues along the way to prevent confusion.

  • We needed to deliver contextual and just-in time information to be a security advocate with Lee so that Gary’s work protecting his systems became easier

We were able to successfully launch what is now known as universal prompt through a coordinated effort among Design, Engineering, Product and Customer Success. 

Additional Impacts

I partnered with Product and Design to present on our experience and outcomes with the HEART Success workshop. As a result, I had additional requests to facilitate this workshop so other teams could collaboratively define their metrics. I trained other researchers and designers to facilitate this workshop, partnering with another researcher to create a workshop kit to make the process easier. This workshop was used for 8+ workshops.

As we continued to evolve our overall product and launched more betas, I became the lead researcher for 7 different projects. I leveraged templates and tools that I discuss in this case study to scale my impact and enable others to do great research.

During this process I started a newsletter to connect the other designers and researchers working on this critical part of the business. I called out key outcomes and linked artifacts and reports. When we had many reports, I conducted a meta-analysis across other’s reports and and facilitated a group synthesis session which culminated in a report to leadership.

This process was exciting to see through as it opened the door to a new phase of our business. We accomplished our goals of delivering an accessible, customizable, improved user experience.