OS-Wizard: Scaling Computer-Using Agents to Specialized Software via Knowledge-Guided Exploration

Zhoumianze Liu, Zichen Ding, Bowen Yang, Zehao Li, Zhenyu Wu +6

Computer-using agents are increasingly capable of performing complex tasks in the digital world. However, high-quality interaction trajectories remain scarce for specialized desktop applications, where limited application knowledge, cumbersome GUI workflows, and incomplete exploration hinder automated collection. We introduce OS-Wizard, an automated trajectory collection framework. It combines tutorial-guided exploration with a hybrid GUI--Code action space, providing application-specific knowledge and simplifying GUI workflows through direct execution of application-native commands. The framework also checkpoints and revisits pivotal states during tree exploration to explore alternative workflows, complementing linear rollouts for broader coverage. We collect 10.9K trajectories comprising 54.9K state--action samples. Fine-tuning open-source base models on this corpus yields a best-performing model that improves performance on a benchmark of specialized desktop workflows by 58.2 percentage points to 64.1\%, surpassing reported human performance of 59.8\% and establishing a new state of the art. The fine-tuned model also improves performance on general desktop workflows using GUI…