Pranjal Aggarwal

We argue that embodied tasks can be framed as interaction with software. We build software that renders an agent's view of a physical environment on a screen and provides controls for acting in it, so a computer-use agent can perform embodied tasks with the same screenshots, keyboard and mouse actions it uses on ordinary software. Using a single agent design and changing only the software, our entries placed first, second, third and sixth in four ECCV 2026 workshop competitions, without any competition-specific agent tuning.

Computer-use agents represent each screenshot as thousands of visual tokens, and most of these tokens are either irrelevant to the next action or repeated across screenshots. We introduce Visual Token Routing (VTR), which trains the vision-language model together with a lightweight spatio-temporal router that decides which visual tokens enter the language model. With just 2,000 training steps, VTR matches the full-resolution model on OSWorld, CUA-World, and video game action prediction while using 2.5-5.8x fewer visual tokens, which cuts inference cost by up to 3.3x.

We introduce Gym-Anything for Robotics, a framework in which coding agents create realistic robotics simulations, so improvements in coding agents translate directly into richer and more faithful simulations. Using an adversarial creation-audit loop, we build RoboWorld, a set of 25 simulations grounded in U.S. GDP data with 25 test and 175 training RL environments. Even the best frontier models fail on every RoboWorld test task, and we show several ways to improve them, including in-context learning, privileged teachers, and code policies that generate demonstrations for training vision-language-action models.

We introduce Gym-Anything-RT, an agentic pipeline that generates visual minigames testing the capabilities AI agents need for real-time interaction, such as acting while the environment changes. The same minigame can be run in real time, slowed down, or paused, which allows controlled comparisons. On 100 generated minigames, GPT-6 Astra (xHigh) passes only 9.0% of the minigames in real time, compared with 75.0% when the environment is paused.

Speed and cost are key barriers to deploying computer-use agents, but current benchmarks run on varying infrastructure that makes speed hard to measure. We introduce cua-speedrun, which provides a uniform virtual machine setup, execution pipeline, and agent interface for measuring the performance, speed, and cost of CUAs across four benchmarks. We find that no single model family is best on all three, and that for some models more reasoning effort can make tasks finish faster.

We introduce Gym-Anything, a framework that turns any software into a computer-use agent environment with a simple make() call. The system includes 200+ real-world applications across 22 occupational categories, generating 10,000+ tasks with checklist-based verification. We also introduce CUA-World-Long, a benchmark of 200 long-horizon tasks evaluating agents on 500+ step interactions.

Computer-use agents (CUAs) hold the promise of performing a wide variety of general tasks, but current evaluations have primarily focused on simple scenarios. We introduce Programming with Pixels (PwP), the first comprehensive computer-use environment for software engineering, where agents visually control an IDE to perform diverse software engineering tasks. We introduce PwP-Bench, a benchmark of 15 existing and new software-engineering tasks spanning multiple modalities, programming languages, and skillsets. We find that when interacting purely visually, CUAs perform significantly worse than specialized coding agents. However, when given direct access to just two APIs -- file editing and bash operations -- performance jumps, often reaching the levels of specialized agents despite having a task-agnostic design.