Validating Your Ideas With the Right Users
A practical guide to creating a high-quality screener
Most agree that consistently getting user feedback to build and optimize products is essential. Whether you are at the beginning stages and have zero users or are optimizing every pixel and have billions of users, gathering data directly from the people who use your product should be ingrained into the building process.
However, I see a common mistake made by many builders. Sometimes it's a post on LinkedIn asking for feedback on a product or sometimes it's a seasoned UX researcher being too lackadaisical in their participant recruiting practice.
This post will be a quick guide to help practitioners get insights from the right people so you aren’t led astray by feedback you shouldn’t be listening to.
What Screeners Are & Why They Matter
Screeners are a set of questions used to find the right users to talk to. Think of screeners as a mini-survey to narrow down the people you talk to.
Imagine you are working for Etsy and want to explore offering small business owners loans directly instead of through your partners. As you explore this new product offering and talk with business owners in the process, you should seek out a specific type of business owner to talk to.
You wouldn’t ask someone allergic to dogs to test a new dog leash. Etsy also wouldn’t ask any business owner without narrowing it down to sellers who would benefit from and use a new loan product.
With that background, let’s examine three high-level principles of a good screener.
Principles of a Good Screener
Targeted: Screeners are a set of questions that you have potential participants answer. Instead of broadly asking any business owner about the potential of a loan, make sure you are talking to people who would be eligible or who the product would actually benefit.
Unbiased: The order and way the questions are asked shouldn't give away the type of person you are looking for. There are professional survey takers who know how to game the system just to get paid.
Clear: Avoid jargon to ensure it is easy to understand.
Keeping these three principles in mind while writing questions will enable you to create a quality screener. Now let’s get tactical and actually write a screener.
Writing a Screener
Let’s use the Etsy example above. Imagine we want to gauge small business owners' perceptions of a new product that provides loans directly from Etsy and is integrated into the seller dashboard.
First, we need to define who we are looking for. For this screener, we don’t need small business owners to be on Etsy already because we believe offering a loan product could bring new sellers to Etsy. We are also looking for small businesses that are already successful ($50,000+ in yearly sales) and have a desire to grow.
Let’s go ahead and start writing screener questions.
Which of the following would you consider yourself?
An employee without key decision-making power [terminate]
An employee with key decision-making power
A passive business Owner [terminate]
An active business owner
In this case, we are ok talking with high-level employees or active business owners. Everyone else will be eliminated from our study.
What was your approximate revenue over the last 12 months?
$0-$10k [eliminate]
$10k-$25k [eliminate]
$25k-$50k [eliminate]
$50k-$100k
$100k-$500k
$500k-$1m
$1m or more
This question eliminates participants who wouldn’t be eligible for a loan. Note: I’m not an expert on Etsy’s sellers, but you may want to cap the revenue a business makes to ensure they are getting “small businesses” in their sample. All you need to do is determine what the cap is and eliminate anyone who answers above it. Similarly, you could also add a question about employee count.
Select all the ways you have tried to grow your business.
Apply for a loan
Grow a social media presence
Partner with other sellers
Partner with influencers
Spend money on ads
Grow my email list
Other:______
I am not trying to grow my business [eliminate]
What you are looking for in this question is that participants are actively looking to grow their business. You may want to prioritize participants who are trying several different ways to grow their business. One pro tip is to randomize the answer options and eliminate participants who select every answer.
Select all the ways to grow you would consider doing in the next 90 days.
Apply for a loan [Must select]
Grow a social media presence
Partner with other sellers
Partner with influencers
Spend money on ads
Grow my email list
Other:______
I am not trying to grow my business [eliminate]
This question ensures participants are open to applying for a loan. If someone wouldn’t consider going into debt to grow, then they probably aren’t the right person to speak with.
Which of the following platforms do you use to sell your products? Select all that apply.
Etsy [50/50 split]
Amazon
Shopify
Facebook Marketplace
eBay
TikTok Shops
Other:______
None of the above
The point is to get a good split of Etsy users vs users of other platforms. Asking the question in this way instead of just asking if they use Etsy (allowing users to self-select) also makes it easier to weed out professional survey takers.
With these five questions (plus questions about contact information if needed), you will have a list of “qualified” candidates to interview about the new Etsy loan product whose opinions and perceptions actually matter when building. Everyone else’s opinions shouldn’t be taken as seriously when building a targeted user-first product.
Where to Distribute Your Screener
Lastly, the question becomes where to distribute a screener to get people to fill it out. Where you distribute a screener depends on the needs of the study. For example, do you want to recruit current users or potential users? Churned users or users who just completed an action or behavior in your app? All these things will determine how you recruit.
With that said, the most common ways to distribute your screener are through the following:
Current user/customer list: You can shoot them an email or create a pop-up of the screener in your product
A third-party panel: There are self-serve tools such as UserTesting and Dscout or more white-glove services such as Ipsos or Kantar. You can select participants who are ready to talk to you from a pool of thousands.
Gorilla recruitment: Join a Facebook group, a subreddit, or go to physical places your customers or potential customers hang out. You can post and distribute your screener there.
There you have it. Next time you do product testing, make sure you are recruiting the right people to get feedback that actually matters.