Jingfan Chen

A benchmark for multimodal agents and world models that interact in generated worlds. Besides a coupled evaluation of agent and world model together, two controlled tracks assess them separately: the agent track fixes the environment’s response rules (1,088 items across 15 task categories), and the world-model track supplies a validated action sequence and checks whether the generated states realize the action effects while preserving what must stay unchanged (400 cases across eight scenarios). Tasks span physical, digital, social and scientific domains, and success is judged against task requirements and execution evidence rather than a model’s own claim of completion.

GUI grounding requires precise identification of interface elements from high-resolution screenshots, but processing dense visual inputs with large vision-language models incurs substantial computational overhead. Existing visual token compression methods typically allocate a fixed budget, ignoring the large variability of visual redundancy across different GUI layouts. We show that a fixed visual budget is suboptimal, as the necessity of high-resolution visual evidence varies substantially across interface contexts. We propose FoveaUI, a dynamic high-resolution budget allocation framework for GUI grounding. The agent first reasons over a low-resolution global view and selectively requests high-resolution crops only when fine-grained evidence is required. FoveaUI introduces three coordinated components that progressively address when to seek high-resolution detail, where to acquire it, and how to leverage it effectively: a balanced training strategy for on-demand resolution allocation, a mix-tool strategy paired with a smooth process-level reward for region and granularity selection, and locality-prioritized attention masking for efficient utilization of acquired high-resolution information. Experiments on GUI grounding benchmarks show that our method maintains strong grounding performance while substantially reducing high-resolution visual token costs.

Process supervision improves perceptual grounding in RLVR-trained vision-language models, but it also incurs repeated verifier queries and reshapes the policy by suppressing exploratory entropy. We show that its benefit and cost are empirically separable: grounding gains emerge early, whereas the exploration cost accumulates with continued supervision, holding the policy’s entropy below that of outcome-only RLVR. This hidden behavioral cost shows up as skill tradeoffs on static benchmarks and, more importantly, as a degraded initialization for subsequent outcome-only RLVR that prematurely limits its reachable ceiling. Motivated by this asymmetry, we propose warmup-and-merge: early-stop the process-supervised warmup, merge only a low-rank component of the warmup task vector into the base model, and continue with outcome-only RLVR under an adaptive on-policy reward that targets the model’s current perceptual failures. Across multiple benchmarks and backbones, our method improves perception while preserving exploration, achieving stronger final performance at a fraction of the verifier cost of continuous process supervision.

A generalist GUI agent pre-trained on about 20.6 million GUI understanding tasks to strengthen element grounding, then fine-tuned on a unified dataset that maps heterogeneous action formats into one shared action space, with strong results across GUI agent benchmarks on multiple platforms.

An automatic pipeline that uses LLMs to annotate what UI elements do by observing how the interface changes after simulated interactions, with LLM-based rejection and verification to filter noisy labels. The resulting AutoGUI-704k dataset improves GUI grounding of vision-language models and scales with data size.

A training-free decoding method that identifies hallucination-related directions in the hidden states of large vision-language models from a small calibration set and steers generation away from them with contrastive decoding.

KGAttack, a black-box attack on recommender systems that uses knowledge graphs to enhance the state representation and candidate selection of a hierarchical reinforcement-learning attacker, which generates fake user profiles under a limited query budget.