This article provides a comprehensive overview of 9 proven testing methods—Interviews, Questionnaires, Heuristic Evaluation, Card Sorting, Co-design, A/B Testing, Think-Aloud Tests, Written Open-Ended Questions, and Usability Studies—known for enhancing product and service development by improving usability, usefulness, and user desirability. Designed as a practical guide for cities, it clearly outlines the purpose, scope, and benefits of each method, helping municipalities select approaches best suited to their needs. It also includes estimated resource requirements, such as the time commitment and number of participants needed for each method, to support effective planning. Additionally, it distinguishes between supervised methods (facilitated by experts) and unsupervised ones (self-administered, such as surveys), providing guidance on the level of engagement required.
Name of Method
Brief description
User research methods can be broadly classified into supervised and unsupervised approaches. Supervised methods involve real-time facilitation, observation, or moderation by researchers—ideal for gathering deep, contextual insights into user behaviour. Unsupervised methods, by contrast, are conducted without direct researcher involvement during the session. These methods scale well and allow participants to engage at their own pace, often anonymously, making them useful for broad outreach and candid feedback.
Supervised Engagement Methods
These methods involve direct interaction, observation, or facilitation by a researcher, offering deep, contextual insights and allowing for real-time clarification and follow-up.
Interviews: User interviews serve as a cornerstone in UX research, particularly in the early stages of design. Skilled interviewers facilitate discussions, probing into topics to grasp insights, motivations, and pain points directly from participants.
Benefits:
Empowering Engagement: Offers participants direct involvement, allowing them to share perspectives and fostering a deeper understanding of user needs.
Influential Insights: Captures valuable feedback early on, shaping product trajectory, direction, and features.
Heuristic Evaluation: Conducted by experts who observe participants carrying out simple tasks based on pre-defined usability parameters. It's similar to Think Aloud tests, focusing on identifying usability issues.
Benefits:
Identify Usability Issues: Helps pinpoint problems in digital products/services that inform UX design processes.
Ensures Usability: Contributes to the overall usability and effectiveness of the product/service.
Co-Design: A participatory method designed to collectively arrive at solutions or decisions with involved stakeholders and experts. It promotes shared ownership and creative problem-solving.
Benefits:
Empowerment: Gives participants a voice, fostering a sense of ownership and inclusion in solution creation.
Collaborative Problem-Solving: Encourages teamwork, leading to more robust and refined solutions through collective effort.
Diverse Perspectives: Brings together varied backgrounds, generating innovative ideas not possible through traditional approaches.
Think Out Loud Test: During this test, participants verbalize their thoughts, feelings, and reactions in real-time as they complete tasks or navigate through a product.
Benefits:
Identification of Usability Issues: Real-time feedback uncovers confusion, frustration, or misunderstandings for timely improvements.
Informing Design Decisions: Understanding user experiences directly guides enhancements to interface, functionality, and overall UX.
Opportunity to Contribute: Participants can contribute perspectives and preferences, ensuring their needs are addressed.
Usability Studies: A facilitator asks participants to perform tasks using specific user interfaces, while an observer takes notes on their behavior and feedback.
Benefits:
Influence Product Development: Participants directly influence product development by providing feedback and suggestions.
Voice User Perspectives: Users can share their experiences, ensuring their needs are considered in design.
Contribute to Improvements: Feedback identifies usability issues, leading to more user-friendly and effective products.
Card Sorting: Participants group individually labeled cards according to criteria that make the most sense to them, helping to define intuitive information architectures.
Benefits:
User Empowerment: Allows participants to directly influence information organization based on their mental models.
Collaborative Design Process: Fosters shared understanding and consensus on information structure between designers and users.
Improved Usability: Creates information architectures that align with user expectations, enhancing navigation and understanding.
A/B Testing: Also known as split testing, this method compares two versions of a product or service (e.g., webpage, app) to determine which performs better based on predefined criteria, often run over weeks to gather statistically significant data.
Benefits:
Informed Decision-Making: Changes are based on empirical evidence rather than assumptions or intuition.
Personalization: Allows evaluation of content tailored to different user segments for maximum relevance.
Data-Driven Culture: Contributes to fostering accountability and transparency in optimization processes within organizations.
Unsupervised Engagement Methods
These methods allow participants to complete tasks or provide input independently, often at their convenience, enabling broader reach, larger sample sizes, and objective data collection without direct, real-time researcher intervention.
Questionnaires: A research instrument consisting of a set of questions (mix of close-ended and open-ended) designed to collect information from respondents remotely.
Benefits:
Flexible Participation: Respondents can participate at their convenience, often anonymously, making it time-efficient.
Uninhibited Expression: Fosters candid feedback as respondents can freely share opinions without immediate in-person constraints.
Written Open-Ended Questions: This method allows users to express thoughts and feedback from their testing experience in a written format, providing a structured way to gather qualitative data without direct verbal interaction.
Benefits:
Barrier and Bias-Free Participation: Allows participants to express themselves freely without the constraints of real-time discussions or performance anxiety, overcoming potential language barriers or biases.
Type/Level of Method
Challenges
Supervised Methods
Directly guided methods, while offering rich insights, come with several challenges:
High Resource Intensity: They are typically more time-consuming and labor-intensive, requiring skilled researchers (interviewers, facilitators, observers) and dedicated time from participants. This translates to higher costs per participant.
Logistical Complexity: Scheduling and coordinating sessions with multiple participants and researchers, securing appropriate testing environments (physical or virtual), and managing recording equipment can be logistically challenging.
Risk of Bias: The presence of a researcher can introduce various forms of bias, such as moderator bias (influencing responses), participant performance anxiety, or social desirability bias (participants giving answers they think the researcher wants to hear).
Limited Scalability: Due to their one-on-one or small-group nature, these methods are difficult to scale for very large participant numbers, which can limit the generalizability of findings.
Analysis Complexity: The qualitative data generated (e.g., interview transcripts, observation notes) is often rich but complex and time-consuming to analyze, requiring careful thematic coding and interpretation.
Participant Recruitment: Finding and recruiting specific, representative users who are available and willing to commit the required time can be a significant hurdle.
Unsupervised Methods
Independent methods offer scale but present their own set of obstacles:
Lack of Nuance and Context: Without a researcher present, it's impossible to ask follow-up questions to understand the "why" behind participant actions or responses, leading to potentially superficial or misinterpreted data.
Dependence on Clear Design: The effectiveness of these methods heavily relies on the clarity and precision of the questions or tasks provided, as participants cannot ask for clarification. Poor design can lead to invalid data.
Data Quality and Interpretation: While quantitative data (e.g., from A/B tests, closed-ended survey questions) can be straightforward, interpreting open-ended responses without context can be challenging and prone to misinterpretation.
Recruitment and Engagement Rates: While scalable, attracting a high volume of responses or ensuring a truly representative sample can still be difficult without strong incentives or broad reach, leading to potential response bias.
Technical Limitations: Reliance on digital platforms and tools means potential for technical glitches, compatibility issues, or limitations in capturing subtle user behaviors (e.g., facial expressions, body language).
Limited Observational Depth: Researchers cannot directly observe real-time struggles, emotional reactions, or environmental factors that might influence a user's experience, which are typically captured in supervised settings.
Problem, Purpose and Needs
Supervised Methods
These methods purpose is to gain deep, qualitative insights into user behavior, motivations, and experiences by exploring the "why." They are needed for early-stage discovery, understanding complex issues, capturing nuanced feedback, and building empathy when rich, contextual data is crucial.
Unsupervised Methods
The purpose of these methods is to efficiently gather broad, often quantitative data from a large user base to validate hypotheses or compare designs. They are needed when statistical significance, scalability, objectivity, and identifying widespread patterns are paramount.
Relevance to Climate Neutrality
Challenges
Thematic Areas
Impact Goals
Issue Complexity
Issue Polarisation
Enabling Condition
Essential Considerations for Commissioning Authorities
Engagement Journey
Governance Models and Approaches
Enabling Conditions
Democratic Purpose
Spectrum of participation
Communication Channels
Actors and Stakeholder Relationships
Actors in UX and usability testing usually include the designers, design researchers, developers and the end users (which may be municipality users, citizens or internal users)
Participant Numbers
Actors and Stakeholders
Participant Recruitment
Interaction between participants
Format
Social Innovation Development Stage
Scope
Time commitment
Supervised Methods
Interviews:
Typical test duration: 30m – 1hr
Required repetitions for conclusive results: 6-12 minimum; 15 is preferred.
Heuristic Evaluation:
Typical test duration: 1-2 hr
Required repetitions for conclusive results: 6-12 (15 preferred) for directly guided assessments; Not Applicable for self-assessments.
Co-Design:
Typical test duration: 1.5-3 hr (including a break)
Required repetitions for conclusive results: Depends on participant numbers to reach a definitive outcome.
Think Out Loud Test:
Typical test duration: 20-30m
Required repetitions for conclusive results: 5-7.
Usability Studies:
Typical test duration: 30-60m
Required repetitions for conclusive results: 5.
Card Sorting:
Typical test duration: 1-2 hr
Required repetitions for conclusive results: 1.
A/B Testing:
Typical run time: 2 weeks minimum - 6 weeks maximum.
Required repetitions for conclusive results: 1 (the test runs continuously).
Unsupervised Methods
Questionnaires:
Typical completion time: 10-15 m
Required repetitions for conclusive results: Minimum 50-100; as many as possible.
Written Open-Ended Questions:
Typical test duration: 1-3 hr
Required repetitions for conclusive results: 6-12 minimum; 15 is preferred.
Resources and Investments
Supervised Methods
Interviews:
Number of Participants per session: 1
Other Resources: Skilled interviewer, a quiet interview space, discussion guide, recording equipment (optional but recommended).
Heuristic Evaluation:
Number of Participants per session: 1
Other Resources: Expert evaluators, pre-defined usability heuristics/parameters, the product/interface to be evaluated.
Co-Design:
Number of Participants per session: Individual sessions: 1; Group Sessions: 3 minimum - 10 maximum.
Other Resources: Facilitator, collaborative workspace (physical or digital), brainstorming tools (e.g., whiteboards, sticky notes, digital collaboration software).
Think Out Loud Test:
Number of Participants per session: 2-3
Other Resources: Functional product/prototype, quiet testing environment, recording equipment (audio/video), facilitator.
Usability Studies:
Number of Participants per session: 2-3
Other Resources: Facilitator, observer, functional user interface(s), testing environment, note-taking materials/software, recording equipment (optional but recommended).
Card Sorting:
Number of Participants per session: 10-5 minimum; 20-30 is considered optimal.
Other Resources: Physical cards or digital card sorting software, labelled content for sorting.
A/B Testing:
Number of Participants per session: 1 (referring to the individual user interaction, but requires a large user base for valid results).
Other Resources: A/B testing software, two distinct versions of the digital asset, analytics tools for tracking performance metrics.
Unsupervised Methods
Questionnaires:
Number of Respondents per session: 1
Other Resources: Survey creation/distribution platform (e.g., online survey tool), a well-designed set of questions.
Written Open-Ended Questions:
Number of Participants per session: 1
Other Resources: A platform for written input (e.g., online form, document), clear prompts/questions.
Typical duration
Resources and Investments
In-house
Step by Step
Supervised Methods
Interviews:
Preparation: Develop a comprehensive set of open-ended questions and a detailed discussion guide to ensure relevant topics are covered and data collection is consistent across participants.
Conducting Sessions: A skilled interviewer facilitates individual, one-on-one conversations with each selected participant, typically in a quiet and comfortable setting, encouraging open dialogue.
In-Depth Exploration: During the discussion, the interviewer actively listens, asks follow-up questions, and probes deeper to uncover nuanced insights, underlying motivations, and specific pain points experienced by the participants.
Data Analysis & Synthesis: Systematically record responses (via notes, audio, or video) and then analyze the qualitative data to identify recurring themes, patterns, and key user needs that will directly inform the product's design.
Heuristic Evaluation:
Define Criteria: Establish a clear set of industry-recognized usability heuristics or predefined parameters against which the product or interface will be assessed.
Task Assignment: Experts provide participants with a series of simple, real-world tasks to perform within the system.
Observation & Assessment: The experts meticulously observe participant interactions and responses, evaluating them against the defined heuristic principles to identify violations or usability flaws.
Problem Documentation: Document all identified usability issues, categorize them by severity, and report findings with recommendations for improvement to inform design iterations.
Co-Design:
Recruit Participants: Bring together a diverse group of relevant stakeholders, including end-users, subject matter experts, and designers, ensuring a range of perspectives.
Facilitate Collaborative Sessions: Guide structured, interactive workshops where participants actively engage in shared problem-solving activities, such as brainstorming, sketching, or prototyping.
Collective Solution Development: Encourage teamwork and open discussion to jointly identify problems, generate creative solutions, and iterate on design concepts together.
Consensus & Decision-Making: Work towards a collective agreement on optimal solutions or decisions, ensuring all involved parties feel a sense of ownership and contribution to the final outcome.
Think Out Loud Test:
Task Setup: Provide participants with specific tasks to accomplish while interacting with a functional product or prototype.
Verbalization Prompt: Instruct participants to continuously articulate every thought, feeling, decision, and reaction that comes to mind as they navigate and complete the assigned tasks.
Observation & Listening: The researcher carefully observes participant behavior and actively listens to their real-time verbalizations without interruption, noting areas of confusion, frustration, or unexpected interactions.
Insight Extraction: Analyze the recorded verbal data and observed behaviors to uncover usability issues, understand user mental models, and inform targeted design improvements.
Usability Studies:
Task Design: Create a set of realistic tasks that participants will perform using one or more specific user interfaces, designed to test key functionalities.
Facilitated Execution: A facilitator guides the participant through the tasks, providing instructions and answering non-leading questions, while remaining neutral.
Observation & Documentation: A dedicated observer (or the facilitator themselves) meticulously takes notes on the participant’s actions, verbal comments, and expressions, documenting any difficulties or successes.
Analysis & Recommendations: Compile and analyze the observed behaviors and feedback to identify usability issues, evaluate the interface's effectiveness, and formulate actionable recommendations for design enhancement.
Card Sorting:
Card Preparation: Create individual cards (physical or digital) each labeled with a distinct piece of content, feature, or concept from the product or website.
Participant Sorting: Instruct participants to independently sort these cards into groups that make logical sense to them, based on their own understanding and mental models.
Group Naming: Ask participants to name the groups they have created, reflecting their categorization rationale.
Analysis & Structure Derivation: Analyze the various sorting patterns across participants to identify commonalities and differences, using these insights to inform and validate the optimal information architecture or navigation structure.
A/B Testing:
Hypothesis & Version Creation: Formulate a clear hypothesis about which version will perform better and create two distinct versions (A and B) of a digital asset (e.g., a web page, button, email subject line).
Randomized Deployment: Randomly present Version A to one segment of users and Version B to another segment of users in a live environment.
Performance Tracking: Continuously monitor and automatically track key performance indicators (KPIs) and user interactions for both versions over a set period, using specialized A/B testing software.
Data Comparison & Decision: Analyze the collected data to compare the performance of Version A versus Version B, identifying the statistically significant winner to implement the more effective design.
Independent (Unsupervised) Methods
Questionnaires:
Questionnaire Design: Develop a well-structured set of questions, incorporating a mix of closed-ended (e.g., multiple choice, rating scales) and open-ended questions to gather both quantitative and qualitative data.
Distribution: Disseminate the questionnaire to a large, relevant target audience, typically through an online survey platform, ensuring easy and convenient access for respondents.
Data Collection: Allow participants to complete the questionnaire at their own pace and convenience, collecting responses automatically via the chosen platform.
Analysis & Interpretation: Compile and analyze the gathered data, looking for statistical trends from closed-ended questions and thematic insights from open-ended responses to draw conclusions.
Written Open-Ended Questions:
Prompt Development: Craft clear, concise, and thought-provoking open-ended questions or prompts that encourage participants to elaborate on their experiences, thoughts, or feedback.
Platform Provision: Provide a suitable platform (e.g., an online form, a dedicated text box within an application, or a simple document) where participants can comfortably write their detailed responses.
Independent Submission: Allow participants to complete and submit their written feedback at their own pace, without the need for live discussion or direct researcher presence.
Thematic Analysis: Collect all written responses and perform a qualitative analysis to identify recurring themes, sentiments, and key insights from the textual data, which can then inform design improvements.
Evaluation
Connecting Methods
Flexibility and Adaptability
Existing Guidelines and Best Practice
Directly Guided (Supervised) Methods
Interviews:
Practice Active Listening: Focus on understanding the participant's perspective, not just waiting for your turn to speak.
Use Open-Ended, Neutral Questions: Avoid leading questions to encourage genuine, detailed responses.
Build Rapport: Create a comfortable, non-judgmental environment to help participants feel at ease and share openly.
Record with Consent: Always obtain permission to record (audio/video) for accurate data capture and later analysis.
Heuristic Evaluation:
Use Multiple Evaluators: Employ at least 3-5 experts to identify a greater number of usability issues.
Reference Specific Heuristics: Clearly apply established usability principles (e.g., Nielsen's heuristics) to guide the evaluation.
Provide Specific Examples: For each identified issue, include concrete examples and explain how it violates a heuristic.
Prioritize Findings: Rank issues by severity and frequency to guide development efforts.
Co-Design:
Ensure Diverse Representation: Include a wide range of stakeholders (users, designers, developers, business) for rich insights.
Establish Clear Goals: Define what problems are being solved and what outcomes are expected from the session.
Use Engaging Activities: Facilitate interactive exercises (e.g., sketching, persona creation, journey mapping) to encourage participation.
Maintain Neutral Facilitation: A skilled facilitator guides the process, manages dynamics, and ensures all voices are heard without bias.
Think Out Loud Test:
Provide Simple, Realistic Tasks: Design tasks that mirror real-world user scenarios and are easy to understand.
Remind Participants to Keep Talking: Gently prompt users to verbalize their thoughts if they become silent.
Avoid Intervention: Resist the urge to help or explain unless absolutely necessary for task completion.
Focus on Behaviors and Thoughts: Observe what users do and listen to why they do it.
Usability Studies:
Recruit Representative Users: Ensure participants closely match the target user demographic.
Create Realistic Scenarios & Tasks: Design tasks that reflect authentic user goals and flows within the product.
Maintain Neutrality: Facilitators should avoid leading participants or influencing their actions.
Document Observations Systematically: Use consistent methods for note-taking and video logging to capture critical moments and quotes.
Card Sorting:
Use Clear Card Labels: Ensure each card has a distinct, understandable label.
Provide Adequate Instructions: Clearly explain the task and the criteria for grouping to participants.
Analyze Consistently: Use reliable software or manual methods to identify common grouping patterns across participants.
Combine with Other Methods: Use card sorting insights as a foundation for further testing, like tree testing.
A/B Testing:
Test One Variable at a Time: Isolate the change you're testing to accurately attribute performance differences.
Define Clear Metrics: Establish specific, measurable goals (e.g., conversion rate, click-through rate) before running the test.
Run Long Enough: Allow sufficient time to gather statistically significant data, accounting for daily/weekly variations.
Ensure Random Assignment: Randomly distribute users to version A or B to avoid bias.
Independent (Unsupervised) Methods
Questionnaires:
Keep it Concise: Respect participants' time by making the questionnaire as short as possible without sacrificing necessary data.
Use Clear, Unambiguous Language: Avoid jargon, double-barreled questions, or leading phrasing.
Test the Survey: Pilot test the questionnaire with a small group before full deployment to catch issues.
Ensure Anonymity (if applicable): Clearly state privacy policies to encourage honest responses
Written Open-Ended Questions:
Provide Specific Prompts: Offer clear, focused questions to guide participants' written responses.
Manage Scope: Keep the scope of questions focused to avoid overwhelming participants or receiving unfocused answers.
Allow Ample Time: Ensure participants have enough time to thoughtfully compose their responses.
Prepare for Thematic Analysis: Plan how you will systematically review and categorize the qualitative text data.
References and Further Resources
Heuristic Evaluation, Usability Testing and Think-out-Loud tests:
https://www.nngroup.com/articles/ten-usability-heuristics/
https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/
Questionnaire Surveys and Open-ended questions:
https://journals.sagepub.com/doi/epdf/10.5005/jp-journals-10021-1104
https://www.nngroup.com/articles/surveys-design-cycle/
Card sorting:
https://www.nngroup.com/articles/card-sorting-definition/
https://www.sciencedirect.com/science/article/abs/pii/B9780081006764000031
Co-design:
https://www.mdpi.com/journal/sustainability/special_issues/0E9KZS61E0