How to approach a problem.
Problem solving activity
The Challenge: Think of your favourite app. In addition to publicly available data, you also have access to internal data sources including web analytics and database of users. How would you go about studying decreased engagement for that app? What would you tell a PM about why you chose these research methods? How would you approach the validation of your insights?
Wealthsimple is an application that allows users to invest on autopilot. This means that users who know little to nothing about investing are now able to enjoy the pay off that they would have otherwise been too risk-averse to approach.
Recently, Wealthsimple has noticed a decrease in their app engagement. Having just joined the Wealthsimple team I’ve been tasked with identifying the reason behind the decreased engagement. The Product Manager has asked for a proposed outline that includes a course of action, reasoning behind choosing particular research methods and a plan of how to validate the insights gathered.
Having start-up experience, I know that utilizing resources properly is important. When you’re working with a small team, it’s not unusual that employees wear many different hats. This means that you cannot afford to be wasting time, money or efforts. Time is expensive and following a lean UX model will allow us to identify issues quickly and cheaply for the benefit of the user and interest of the company. This is important because it allows us to gather information and make fast decisions. Working in an Agile framework ensures that we’re moving quickly and funneling data-driven decisions into each iteration of our product. At Wealthsimple, our core concern is our users. If we’re not providing the best user experience we can, then we need to fix that.
My approach will be to first look internally to see if we can identify the problem without having to bring in users for testing. Working with other people’s schedules can take time and planning. If we’re able to locate an error in our system, we can quickly make the changes ourselves.
1 It is assumed that the parameters defining “engagement” have been outlined by Wealthsimple
2 It is assumed that engagement is clear and measureable
3 It’s assumed that decreased engagement has been measured over time
4 It’s assumed a target user/persona has been previously compiled by Wealthsimple
5 It’s assumed we’ve received consent by the user to contact, test and survey them
6 It’s assumed that usability improvements will increase/grow engagement
I’ll begin by looking at our internal analytics platform to see if there are any identifiable trends in behavior. Behavior being things like navigation paths, marketing responsiveness, frequency of app use, location of application access, clicks, and time between decisions. Perhaps I can identify if there’s a section of the app that has a high rate of user exit or bounce rate. This could allow us to hone in on the problem area. Simultaneously, I’ll reach out to customer support to see if there have been any tickets opened or complaints from users that could lead us to the problem. Web and Mobile Development will also be contacted in hopes that maybe there are some error logs that could raise a flag.
Next step is to log into our user database to see if there is a particular user segment that is showing a decrease in engagement over time. I’m looking for commonalities in areas such as their age, location, investment bracket, gender, or any other socio-demographic characteristics that could help in building a user proto-persona. The purpose of a proto-persona is to compile an outline of target user needs and behaviors. It allows us to put a face to our users for us to reference. It’s solely meant to act as a benchmark to give us a place to start researching. It also allows us to constantly be thinking about our user, rather than selfishly designing for ourselves.
I’ve begun speaking with the marketing, product and strategy departments to see if there have been any recent changes in initiatives or strategy within the past few months. If there have been any changes, this may have negatively affected our target demographic, resulting in a drop of engagement. In conjunction with the data and marketing teams, I’ll want to better understand who our target user/primary user is. I’m interested to know the baseline engagement of the user; to better understand the extent of the lowered engagement we’re experiencing. It would be helpful to understand how often this segment uses the app, how often they deposit money, what their income is, what is their stability of input (meaning, what is their ratio of deposit to withdrawal frequency,) what is their rate of referral, where do they live, and what types of jobs do they have.
05 WHERE TO START
I haven’t seen any red flags internally. This leads me to the assumption that our users might have had a shift or change in their lives which is causing them to engage less with our service. I begin to do research into recent changes in the market or competitive landscape. Has there been a decrease in interest rates? Did a corporation employing many users announce they’re closing? Did a recent housing market crash happen causing users to be more careful with their money? If we can answer these questions, it may give context for our drop in user engagement. It may also prove that the reasoning for that decreased engagement isn’t for fault of our own, but rather uncontrollable external circumstances. If this is the case, then we could make some assumptions about the needs of our users and pivot our product offering strategy for the affected segment. I dive deeper into journals and newspaper articles but don’t see anything that would raise suspicion.
I begin to also conduct a competitor analysis. Have competitors upped their offerings and experience for users, causing them to tap into our user base? Are there any new players in the market who are offering our users a better experience?
I sit down with the Product Manager to discuss my findings. I outline how I plan on working with our users to identify the cause of the decreased engagement. He asks me to outline my objective for user testing.
06 THEN WHAT?
Identify why users are engaging less with our app
Are there usability or design issues with our app?
Do users know, use and understand how to use our app?
Are there other apps providing a better experience for our user?
As we’re an investing application, do users tend to use it once to deposit money and feel comfortable to not check it frequently/leave it alone?
07 PRIMARY RESEARCH
It’s important to go for the low-hanging fruit. What is the easiest way to reach out to a large sample set of our users in a quick, easy and insightful way? My recommendation would be to compile a survey, pull the list of users showing decreased engagement (from our user database) and reach out to them directly through our app. For example, I would work with the development team to create this survey and send a push notification saying something like “We noticed you haven’t used Wealthsimple in a while! We’d love it if you could fill out this survey so we can make our app better for you.”
By conducting 1:1 interviews, it allows us to understand the (attitudinal) qualitative reasoning behind the behavior we looked at in the internal analytics platform. My goal is to understand the “why” behind the “how much/many.” This will give us a more well-rounded understanding of our user’s psyche, helping us identify their needs and behaviors. It also allows us to benchmark against the proto-persona we compiled and fill in any gaps we might have missed.
I reach out directly to a selected few of our long time users who have shown decreased engagement, over time, to see if they’d be willing to sit down with me for 30-60 minutes. My plan is to understand their history of interacting with our service, their experience and outcomes of using our service, what changes have come up in their life recently, any shortcomings in our offerings and their thoughts of how we could make their experience better. I’d also like to understand as a user, what is it that they require in order to feel good while using our application. Are we giving them the information and feedback that they need? What are the pain points?
07.02 1:1 INTERVIEWS
Un-moderated user testing allows me to understand what it is that they’ll actually do (compared to what they said in 1:1 user interviews.) This gives further insight into their behaviour by watching their interaction with the app.
I’m looking to understand how users are interacting with our application. What are areas of the application that their drawn to and which do they avoid? Are there any areas they’re stuck on or confused by? What do they see? How do they feel? What are they thinking? This allows us to better understand how our users interact with the application and if it aids them in the way we believe it to.
I’ll ask them to complete 4 tasks:
1 Go through the on-boarding process:
Does this instill confidence in the user to invest? Do they understand how investing works? Do they feel comfortable investing their money into this application?
2 Deposit money into their investment account
Do they know how to set this up? Do they feel comfortable doing this? Is this function clear and easy to do? Do they know how to do this after watching the onboarding?
3 Find the breakdown of their investment portfolio
Do they know where to find this? Do they know what this means? Is it easy to locate?
4 Check on their investments / ROI
Do they understand how investments work? How would they feel if they’ve lost money? Do they understand the language used on this page?
Follow up questions for the survey included in the un-moderated test:
How often would they check on their investments?
What would remind them to come back or to open the app?
07.03 UNMODERATED USER TESTING
I speak with the Product Manager about implementing diary studies with some new users across different user segments as an ongoing practice. This attitudinal approach will provide further insight into how users interact with our application over time and in their own environment. We will be able to understand their initial thoughts and feelings up to present day.
I’m asking them to:
1 Record aspects of their lives that are relevant to our application
2 Describe their thoughts and feelings towards our application over time
By taking this attitudinal approach, I want to test why users haven’t been interacting with our app. Are we not engaging them enough? Do they not feel the need to constantly check our app? When do they tend to use our app?
07. 03 DIARY STUDIES
After compiling all learnings, I create a detailed user persona that represents this unengaged user segment. This will help us to better understand the needs of our user. What are their personality traits? What do they require to feel happy or accomplished? What are their needs? I include the pain points identified from research methods, as well as any opportunities that arose while speaking with users. This leads to my hypothesis of the reason behind the decrease in user engagement.
I work with the UX designers to create prototypes that incorporate the identified changes. The prototyped changes are centered around the elimination of pain points by our users, allowing us to validate if our insights are correct. We test this by interviewing our same selected users again to see if we’ve addressed the issues.
We make note of the feedback provided by users in the second round of user testing and make any new additional iterations. I then take the final iterations back to the Product Manager. We discuss the proposed changes and filter out those that are unfeasible for the business and users. We decide to test the new iterations against the old with a selected user base that includes our engaged users. We want to see how these changes might affect the usability for our entire user base.
Once we’ve identified our solution to the decrease in engagement, we work as a team (marketing, product, web development, and any other customer-focused departments) to create the changes and take it to market for our users. Based on our testing, we should have a good indication of an expected outcome once the change is fully implemented. We will continue to monitor behavior through the internal analytics platform in order to measure the actual vs expected change (based on our interviews) in engagement over time, allowing for early warning indicators as to whether or not additional research/changes will be needed.
08 INSIGHT VALIDATION
I’d like to bring in our original unengaged users for a 1:1 to speak with them about the new changes. I’d like to understand how they’ve been feeling with the new changes. Do they find themselves using the app more? Have we created a better experience for them? What other questions, comments or concerns do they have? How, in the future, can we continue to better their lives?
Moving forward, I’d like to conduct intermittent interviews throughout our iterations in order to keep our users involved through different steps in the design process. This will allow us to be constantly thinking about designing for the user and addressing any potential problems that arise early on in the process. I believe creating a community revolving around design and users will help us at Wealthsimple continue on our journey towards helping users invest (confidently) on auto-pilot.