Participant ratings across studies 🌟

Participant Ratings lets your team evaluate research participants at the end of each session using standardized, customizable rating questions. Ratings are aggregated across all studies to build a participant quality score for everyone in your pool; giving you a reliable signal of who shows up, engages, and completes studies/tasks over time.

How it works

After a session, facilitators are prompted to answer your organization's rating questions for each participant. Their answers feed into that participant's historical quality score, which is visible in the pool and can inform future recruitment decisions.

Ratings are separated by facilitation type; Remote and In-Person, so you can tailor questions to each context.

Setting up rating questions

Who can configure questions

Participant Ratings is a one-time org-level setup, restricted to account owners and admins. The questions you define apply across every study in your organization and cannot be customized per study.

Important: Once ratings start coming in, question wording should not be changed. Because ratings are aggregated across your entire pool, rewording a question would invalidate all prior responses and break each participant's historical quality score.

Question limits

Each facilitation type (Remote and In-Person) supports up to 3 rating questions. The counter in the top-right corner of the page shows how many you've used (e.g., "Add question (2/3)").

The anchor question

The first question on each tab is special, as it anchors the participant quality score and cannot be deleted. It must remain in position 1. This ensures consistent scoring across your pool.

Adding a question

1. Navigate to Home > Participant Ratings in the left sidebar.
2. Select the tab for the facilitation type you want to configure (Remote or In-Person).
3. Click + Add question in the top right.
4. Fill in:
   • Question text: what the facilitator will be asked (e.g., "How engaged was this participant during the session?")
   • Answer type on a 1–5 scale
   • Low label: the label for the bottom of the scale (e.g., "Disengaged")
   • High label: the label for the top of the scale (e.g., "Highly engaged")
5. Click Save.

Editing a question

Click the Edit button next to any question to update its text or labels. Questions that already have responses collected will show a "Has ratings" lock indicator.  Take extra care when editing these, as changing the wording affects how historical responses are interpreted.

Deleting a question

Optional questions (questions 2 and 3) can be deleted using the Delete button. The first question is required and cannot be removed.

Remote vs. In-Person

The Remote and In-Person tabs each maintain their own independent set of rating questions. The correct set is automatically shown to facilitators based on the study plan's location setting, so facilitators always see the questions relevant to their session type.

Tips

• Set up questions before running studies. Once participants are rated, changing question wording disrupts score continuity across your pool.
• Keep the anchor question stable. It drives the quality score, so its meaning should remain consistent long-term.
• Use the optional 2nd and 3rd questions for secondary signals, like task completion, technical issues, or communication quality that don't need to anchor the main score.