Surveys are harder than they appear on the surface. If you’re working in product design or UX, and don’t have the luxury of a researcher on your team, here’s how to improve the next one you write.
Notes taken from ‘Making sentiment surveys practical’, a talk given by Judd Anti (Director of Research at Airbnb).
When you’re writing a sentiment survey that is going to inform product or design decisions…
Make it short, sweet and on brand.
- Shorter words
- Simple constructions (ie say things in the most straight forward way possible)
- Use brand voice, as the survey will inevitably form part of a person’s overall experience with your product.
Forget that academic BS you learnt at uni and avoid overly formal or scientific language. It hurts comprehension and also may effect the answer you’re going to get because people feel obliged to answer in a way that matches the question, known as a framing effect.
At the same time, don’t make it too colloquial or informal. Turns of phrase might get in the way of a clear question.
There’s a sweet spot in there somewhere. I’ve found this to be true in pretty much any deliverable that people are going to read.
Get specific… NPS ain’t going to cut it
NPS is a proxy for brand perception. If you want surveys to inform product and design decisions, you need specific feedback that will yield actionable insights.
- Focus on a specific experience - e.g, the last time they used product.
- Get close to the context of use by asking about the experience right after it happens, and avoid memory bias.
- Identify the most granular unit i.e. zone in to a really specific aspect of the experience or interface.
Consider composite metrics
Make a metric out of multiple questions:
- Figure out the things that are crucial to delivering the experience you have in mind
- Come up with a question that will address each one
- Combine them in to a single metric
- Average them in to mean / weight certain questions more heavily / whatever Google’s HEART framework springs to mind as an example.
Bottom Up Sentiment (BUS) to make sure your survey is for real
Before you can write an effective survey, you first need to do some ground work to understand the key themes of an experience and the language real people use to describe these key themes. For a buyer on Gumtree, it would be ‘finding things to buy’, ‘negotiating’ and ‘exchanging’. DON’T let executives define the questions that are being asked or language they are being asked in.
- Qualitative feedback Gather your inputs.
- Thematic analysis Distill the key themes that came out of qual research.
- Survey generation Make the damn survey.
This is a pretty new approach for me. I didn’t realise that steps 1 & 2 existed. The logic seems solid though.
General tips for surveys
- When using scales, never go beyond 1 - 5
- i.e. don’t have a 1-10 scale. They don’t fit well on mobile screens and put more cognitive load on people without actually making the answers more accurate.
- Aim for 4-6 themes in the survey, with 1-2 questions per theme. Your survey should end up < 10 questions