Benchmarking Longitudinal Survey
UX Research | Quantitative Research | Survey Design
Tools Used
Project Scope
My Role
Figma, Qualtrics, Condens, Monday.com
13 week timeline before release
UX Research and Project Lead
WHAT IS THE PROBLEM?
The Beam Design System at Viasat had existed for designers for about 3.5 years, for developers for 1.8 years, and as a full-stack design system for 1.75 years. With Beam came an entirely new structure for producing designs for Viasat. This meant a change of processes and new considerations for product teams. The Beam team was now asking important questions about user experience and user satisfaction with using Beam and wanted to measure their services to diagnose high-level issues to probe into during major version releases.
How has users’ experience been with the Beam Design System?
THE SOLUTION AT A GLANCE
A longitudinal survey which is built as a robust quantitative benchmarking study. This is intended to be used for every major version release, reaching 300+ users across 5 stakeholder groups that interact with the Beam Design System.
MY PROCESS
Informational Interviews
Service Map
Drafting a longitudinal study
Feedback
Survey Draft
Survey Release
Analysis
I. INFORMATIONAL INTERVIEWS
Starting off this project, I was new to design systems. To familiarise myself with the concept, I scheduled informational interviews with the Beam Team to understand the scope of the design system and how it is used across stakeholders. I conducted 6 informational interviews with the team to understand Beam’s use by designers, developers, product managers, and product owners. I prepared a small interview script to understand-
Who interacted with Beam and how
Beam Design System consists of reusable components and guidelines that help build products for the company-
Beam Design System
Just like a lot of companies have a design system that provides products with a consistent visual language and provides a set of standards across products, Beam Design System does so for Viasat. It consists of various elements that promise usability and standardization across products by providing consistency and brand identity through its visual language, increased accessibility, and velocity from design to development. It is currently being used across various digital products in the company by-
Designers
Use tokens and design components on Figma to work on wireframes
What a successful interaction looked like
</>
Developers
Use the component library to convert the designs from Figma to code
Frustrations for each stakeholder
Product Managers
Head teams that interact with Beam as they are the visionaries behind products
Product Owners
Head teams that interact with Beam as they are the technical owner of the product
II. SERVICE MAP
After my informational interviews, I worked on a service map of the design system to visualize its use. This helped me lay out who interacts with what parts of the design system and at what stages.
III. DRAFTING A LONGITUDINAL STUDY
.After gaining a good understanding of the design system, my next challenge was identifying an appropriate research method for my study. With the aim of benchmarking,
I chose to design a survey that reached a variety of stakeholders and over 300 users
It would provide valuable quantitative insights and is cost and time-efficient for the organization
An added benefit is being able to compare insights from the previous versions, allowing for benchmarking and helping the Beam team move closer to the team’s mission statement of promoting brand consistency, usability, and accessibility, and assisting in the production of better designs.
The implications of results would look different for different users since they interact with Beam differently and measure their satisfaction using different variables. For example, more design components would mean success for designers, whereas for product owners, it may mean increased consistency across the designs of their products. And so, I started my process of drafting questions on Figjam for the four stakeholder groups. For each of the stakeholders, I targeted all points of interaction-
IV. FEEDBACK
After completing my first survey draft, I engaged in iterative feedback cycles with my manager and another senior researcher. The aim was to understand if my survey was drafted as-
A good quantitative research study. This meant checking for good research practices like not asking leading questions, ensuring the language of the questions was clear to the stakeholders, and minimizing the number of questions being asked to reduce survey fatigue, among others.
A good longitudinal study that can be used for the upcoming major version releases. This meant asking questions that allow for comparison of survey findings across versions to understand how the metrics have changed over time.
V. SURVEY DRAFT
Once we were all satisfied with the quality of the survey, I presented it to the Beam team to ensure there were no topics left that they wanted user feedback on. I also wanted to conduct a few pilot sessions with the team and leave room for potential improvements and changes in the survey.
The survey logic was coded in Qualtrics and had the following sections-
Screener (logic-based separate paths were defined here)
Onboarding
Beam Tools
Beam Support
Additional Feedback
On the last day of my internship on September 8th, the survey was released! It was sent to all stakeholders over email and Slack (Beam Support channels).
VI. ANALYSIS
The survey results came in after my internship ended. It received 49 responses- 23 developers, 20 designers, and 6 Product Managers and Product Owners
Overall, the satisfaction and confidence in using Beam are high across all stakeholders
All stakeholders considered the onboarding experience to be excellent and great
However, most developers mentioned they did not know the ‘Way of Working’ document was provided to them to support onboarding
Most developers never and designers rarely make contributions to Beam due to it not being prioritized by the product lead, the design system being sufficient for their use, or not knowing how to contribute
Most Product Managers and Product Owners sometimes or rarely prioritize Beam version updates in their sprint planning since it is not a priority, they need guidance, they rely on others to do it or it is too confusing to understand
All stakeholders never or rarely participate in Beam Office hours due to not being invited, not being available, or being busy
PROJECT IMPACT
These insights provided the Beam Team with calls to action on their current procedures, especially onboardin and feedback, allowing them to improve 3/5 KPI in the use of Beam.
As the first-ever longitudinal study for the design system, this survey now provides a way for the Beam Team to track metrics over time across different interaction points. They can measure user satisfaction and confidence among other parameters, and compare these across version releases to track their progress.
PROJECT LEARNINGS
Design Systems are COMPLEX! Working with the Beam Design System gave me first-hand exposure on learning how to juggle the needs of different stakeholders- Beam team, Researchers, designers, developers, product managers, and product owners!
I learned that there are fundamental differences in designing a regular study vs a longitudinal study. Longitudinal studies are meant to be replicated. There need to be quantitative measurements that can be compared year after year. And so, they need to be catered to carefully while designing.
Learning from collaboration was one of my key takeaways. Not only was one of my main learnings through working with the Beam Team in understanding design systems, but I also gained valuable insights by collaborating with senior researchers.
Trust the iterative design research process. In the middle of my iterative feedback cycles, there was a point where everything felt wrong. However, it is important to step back, take a breath, and get back to the process with a lot of trust. As one of my seniors very accurately stated,