0

Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes

Computer use agents (or "agents") are generative AI that automates actions within user interfaces from user commands. Current research focuses on training and evaluating the underlying models, leaving these agents' user experience (UX) understudied.

Open
Preview
Year
2025
ArXiv
arxiv.org/abs/2510.04452
Hosting
Full text hostedCC-BY-4.0

Cite

Notes

Only stored in your browser.

Attribution

Abstract & full text
arxiv.org/abs/2510.04452CC-BY-4.0
TL;DR
Semantic Scholar
Attribution policy →

Abstract

Computer use agents (or "agents") are generative AI that automates actions within user interfaces from user commands. Current research focuses on training and evaluating the underlying models, leaving these agents' user experience (UX) understudied. We conducted two studies to understand the design space of agent UX (RQ1) and the support required to prototype it (RQ2). First, we develop a taxonomy of design considerations for agent UX, comprising 21 subcategories of UX considerations. Then, through a requirements elicitation study with 12 participants---including six agent experts---we identify five Activities and six Desired Capabilities needed in tools prototyping agent UX. Informed by these results, we created AgentUXlab, a design probe that enables developers to design agents with different UX approaches for a website and evaluate those experiences by executing prototypes in a browser. From a user study with 14 participants, we elucidate tooling insights and derive design implications for agent UX prototyping tools.