Jul 12th, 2010 |
Today’s post is brought to you by Web 2.0 Expo New York speaker Laura Klein whose session Combining Qualitative Research, Quantitative Analytics, and Vision to Improve Design covers “qualitative and quantitative methods to learn from users, generate great design ideas, and validate your product decisions.”
Come see Laura this fall by registering today. Use discount code webny10scm4 to save 25% on all conference passes.
I’ve been talking to a lot of startups recently, and it seems like they’re all saying the same thing: “We use metrics to help us make the right product decisions.”
Don’t get me wrong. Metrics are an incredibly useful tool, and every company should be A/B testing and measuring the impact of their new features and product changes. But if you’re relying solely on metrics to steer your product development decisions, you’re going to cost yourself a lot of extra time and money.
The thing is, metrics are great for measuring things: acquisition, retention, revenue, etc. They’re fantastic for telling you what your users are doing. But what they can’t tell you is why.
Let me give you an example. Imagine that you’re looking at your metrics dashboard one day, and you realize that you have quite a lot of drop off in the middle of your registration flow. You now have a very clear goal: decrease that drop off so that more people who hit your landing page will become registered users.
Here is a wildly incomplete list of the things that might be causing the drop:
- People don’t have the information necessary to move on
- People don’t want to give you the information you require because of privacy issues
- People don’t understand what you’re asking for
- People can’t find the button to move them to the next step in the process
- People get bored and wander off because there are too many steps
- People get confused about what to do next
How do you know which of these things is causing the problem? Or what if it’s some completely different problem that you haven’t even thought of yet? All the metrics and measurement in the world will not tell you exactly which problem it is or how to fix it. Metrics can only tell you that a problem exists and whether you’ve solved the problem, but you can spend a lot of time implementing random “fixes” and then testing them, only to find out that you haven’t moved the needle one iota.
So, what’s the answer? You’re going to have to add a little qualitative research and a dash of design vision to your quantitative data in order to get really good results.
Qualitative research, which involves talking to users, observing them using your product or other people’s products, and general customer development, can tell you why users are doing what they’re doing. By watching a few users struggle through your registration flow, you can pretty quickly determine what specific problems are causing users to drop off. By talking to users and asking intelligent questions, you can take most of the guess work out of discovering what they hate.
Of course, identifying the problem is only half the battle. The other half is figuring out the right way to fix it, and that’s where vision comes in. Somebody has to come up with some sort of clever solution that will solve the users’ problems without causing new ones. You still may have to try a few different solutions, but at least you will know you are working on solving the right problem.
Once you’ve come up with a fix and implemented it, metrics are the only real way to measure whether it worked. This can create a great feedback loop for evaluating your research and design decisions. By incorporating all three methods, qualitative, quantitative, and vision, into your development process, you can dramatically decrease the number of iterations it takes you to get to something your users truly love.
Interesting in learning more about this strategy and getting some tips for implementing all three methods? Come to my session: Combining Qualitative Research, Quantitative Analytics, and Vision to Improve Design.
~~
Laura Klein is the Principal at Users Know. You can read her blog at http://usersknow.blogspot.com or follow @lauraklein on Twitter.
Laura, good insight on testing. However, I tend to think testing should be used more for optimizing, not fixing problems. I believe fixing problems, like the one you have pointed to (sharp drop off in the funnel process) should be subjected to investigation. Optimizing a funnel process however does call for testing which helps me understand user preference, usability, and flow.
just my 2cents.
Steve