Evaluating User Experience in Innovation Tools

Traditional usability tests often treat user experience factors in isolation, as if different factors like usability, trust, and satisfaction are independent of each other. But in reality, they are deeply interconnected. By analyzing each factor separately, we miss the big picture - how these elements interact and shape user behavior. This is where Structural Equation Modeling (SEM) can be incredibly helpful. Instead of looking at single data points, SEM maps out the relationships between key UX variables, showing how they influence each other. It helps UX teams move beyond surface-level insights and truly understand what drives engagement. For example, usability might directly impact trust, which in turn boosts satisfaction and leads to higher engagement. Traditional methods might capture these factors separately, but SEM reveals the full story by quantifying their connections. SEM also enhances predictive modeling. By integrating techniques like Artificial Neural Networks (ANN), it helps forecast how users will react to design changes before they are implemented. Instead of relying on intuition, teams can test different scenarios and choose the most effective approach. Another advantage is mediation and moderation analysis. UX researchers often know that certain factors influence engagement, but SEM explains how and why. Does trust increase retention, or is it satisfaction that plays the bigger role? These insights help prioritize what really matters. Finally, SEM combined with Necessary Condition Analysis (NCA) identifies UX elements that are absolutely essential for engagement. This ensures that teams focus resources on factors that truly move the needle rather than making small, isolated tweaks with minimal impact.

AI changes how we measure UX. We’ve been thinking and iterating on how we track user experiences with AI. In our open Glare framework, we use a mix of attitudinal, behavioral, and performance metrics. AI tools open the door to customizing metrics based on how people use each experience. I’d love to hear who else is exploring this. To measure UX in AI tools, it helps to follow the user journey and match the right metrics to each step. Here's a simple way to break it down: 1. Before using the tool Start by understanding what users expect and how confident they feel. This gives you a sense of their goals and trust levels. 2. While prompting  Track how easily users explain what they want. Look at how much effort it takes and whether the first result is useful. 3. While refining the output Measure how smoothly users improve or adjust the results. Count retries, check how well they understand the output, and watch for moments when the tool really surprises or delights them. 4. After seeing the results Check if the result is actually helpful. Time-to-value and satisfaction ratings show whether the tool delivered on its promise. 5. After the session ends See what users do next. Do they leave, return, or keep using it? This helps you understand the lasting value of the experience. We need sharper ways to measure how people use AI. Clicks can’t tell the whole story. But getting this data is not easy. What matters is whether the experience builds trust, sparks creativity, and delivers something users feel good about. These are the signals that show us if the tool is working, not just technically, but emotionally and practically. How are you thinking about this? #productdesign #uxmetrics #productdiscovery #uxresearch

A proposed qualitative evaluation framework for Generative AI writing tools: This post is my first draft of an evaluation framework for assessing generative AI tools (e.g. Claude, ChatGPT, Gemini). It's something I’ve been working on with Ryan Low — originally in the interest of selecting the best option for Rotational. At some point we realized sharing these ideas might help us and others out there trying to pick the best AI solution for your company's writing needs. We want to be clear that this is not another LLM benchmarking tool. It's not about picking the solution that can count the r's in strawberry or repeatably do long division. This is more about the everyday human experience of using AI tools for our jobs, doing the kinds of things we do all day solving our customers' problems 🙂. We're trying to zoom in on things that directly impact our productivity, efficiency, and creativity. Do these resonate with anyone else out there? Has anyone else tried to do something like this? What other things would you add? Proposed Qualitative Evaluation Criteria 1 - Trust and Accuracy Do I trust it? How often does it say things that I know to be incorrect?  Do I feel safe? Do I understand how my data is being used when I interact with it? 2 - Autonomous Capabilities How much work will it do on my behalf? What kinds of research and summarization tasks will it do for me? Will it research candidates for me and draft targeted emails? Will it read documents from our corporate document drive and use the content to help us develop proposals? Will it review a technical paper, provided a URL? 3 - Context Management and Continuity How well does the tool maintain our conversation context? Not to sound silly, but does the tool remember me? Is it caching stuff? Is there a way for me to upload information about myself into the user interface so that I don’t have to continually reintroduce myself? Does it offer a way to group our conversations by project or my train of thought? Does it remember our past conversations? How far back? Can I get it to understand time from my perspective? 4 - User Experience Does the user interface feel intuitive? 5 - Images How does it do with images? Is it good at creating the kind of images that I need? Can the images it generates be used as-is or do they require modification? 6 - Integrations Does it integrate with our other tools (e.g. for project management, for video conferences, for storing documents, for sales, etc)? 7 - Trajectory Is it getting better? Does the tool seem to be improving based on community feedback? Am I getting better at using it?