Building Lasting Relationships with Members of Your User Panel

Getting New Members to Engage Right Away

Many companies let users join user panels just for the sake of it, then invite them to participate in their next research project, which might be months away. If you engage with new members too much later, you’ll lose precious momentum and your new members might lose interest. When you have a continuous-research practice, with frequent light touchpoints with your users, you’ll always be able to recruit participants for specific interviews.

Recruiting participants for interviews sustains the momentum. Plus, it gives you the opportunity for a personal welcome, lets you find out anything you might have asked in a profile survey, and minimizes no-shows. The earlier the interview appointment, the better.

No-shows are a real problem, and you can avoid them by booking your interviews early, sending several reminders, and choosing the right channels for your interviews. One study at Airtime faced many no-shows by senior people because they weren’t tech savvy and were uncomfortable using Zoom, so they switched to phone calls. For another research project, having screen sharing was crucial, so the researchers dedicated an extra 15 minutes to each session to call respondents, then set up the Zoom connection together on the phone.

Preparing for the First Session

The first interview session is an excellent opportunity to set the tone of your new relationship, find out a lot of personal things about your new panel member, and of course, get the feedback you need for your ongoing research. If you’re running a collaborative research practice—and you should be to gain the numerous benefits I outlined in my blog post “How Collaborative User Research Breaks Down Organizational Silos”—you’ll include other stakeholders from the product team as observers.

1. Before the session, define the key learnings you want to get out of the session, then share them with your stakeholders so they can provide feedback on them or suggest their own. Prioritize them as necessary.
2. It is an excellent practice to ask participants to write down their answers to these key questions in advance. Comparing their assumed answers to their actual answers after the interviews can uncover important assumptions and biases within the product organization.
3. Distribute tasks. Simply sitting in a meeting is much less effective than actively participating. For example, the user researcher could lead and facilitate the sessions, the product manager could take notes, and the designer could note down metacommunications from the participant. Alternatively, the observers could all write down their own key learnings from what the participant says. It would be interesting to compare whether these key learnings match and, if not, why not. Then everyone can ask questions at the end of the session, during an open Q&A.

Contact Frequency

Professional panel providers recommend engaging with your user-panel members at least once a month. On the other hand, theirs are panels for which respondents get paid for participating. In my previous column, I explained why I avoid paid-panel providers. They usually make recruitment a transactional, one-off event; their filters are not sufficiently granular; and they cost a lot. Of course, these are companies with deep, user-research cultures, who have already built a panel of trusted advisors. One such company, a leader in software development for the global music industry, rotates quarterly Net Promoter Score (NPS) respondents to avoid burnout. As a result, they ask their NPS respondents to participate just twice a year at a maximum.

There is no silver-bullet answer to the ideal contact frequency, so let’s take a more generic approach. A top-down point of view looks like this:

Ideal contact frequency for you to contact each panel member = [frequency of your releases that need testing] * [number of people on your panel] / [minimum number of people you must ask for each release] * [willingness to respond as a %]

For example, if you do monthly releases and want to ask ten people to participate in your research for each release, you have 100 people on your user panel, and you need to ask 20 people to get ten to participate, the equation looks like this:

[monthly releases] * [10 you want to ask] / [100 full panel population] * [50% response willingness] = monthly * 10 / 100 * 50% = you contact each panel member every five months

A bottom-up approach to getting a sense of panelists’ willingness to participate is to communicate directly with each panel member. Or you could assess activity levels on channels such as your product forum. You can grade each person’s willingness on a scale of three or five and contact the more willing participants more frequently.

Ideally, the top-down approach—that is, recruiting on the basis of what you need—and the bottom-up approach—which depends more on what panelists need—should meet in the middle, so you can contact each panel member as often as is convenient for them, while getting enough feedback for each of your releases.

Channels That Build Community

In addition to direct, one-on-one contact channels such as video interviews and surveys, consider complementary community channels as well. Community forums or chat channels such as Slack encourage interactions between your panel members. This could enable you to uncover insights that are unknown unknowns to you.

Chats are great for more synchronous discussions and nearly real-time interactions, while forums provide an asynchronous experience. These discussions tend to be more structured because of their clearly delineated topics, up- and down-voting capability, and better search. Tools such as Zenloop use Natural Language Processing (NLP) to analyze and classify forum insights. NLP algorithms are good enough to indicate sentiment or the key themes of a discussion. You could subsequently organize sessions with users for deep dives into key themes. You would need about 100–150 active participants in a forum for participants to generate meaningful, but not overwhelming amounts of content.

Don’t forget to mention these communities during your first interactions with new panel members, so they can immerse themselves right away.

Conclusion

If you want to make the most of your user panel, you’ll need to treat each member as a valued friend. Personalized, albeit resource-heavy relationships form the basis on which you can build your product advisory panel. If you can set up an automated recruitment pipeline and focus your efforts on building these key personal relationships, you’ll be able to understand your market on the go, in perpetuity. This is the equipment you need to build the best product in whatever sector you work. Good luck!