A benchmark for multimodal agents and world models that interact in generated worlds. Besides a coupled evaluation of agent and world model together, two controlled tracks assess them separately: the agent track fixes the environment’s response rules (1,088 items across 15 task categories), and the world-model track supplies a validated action sequence and checks whether the generated states realize the action effects while preserving what must stay unchanged (400 cases across eight scenarios). Tasks span physical, digital, social and scientific domains, and success is judged against task requirements and execution evidence rather than a model’s own claim of completion.
@article{xgen2026dualexam,title={Dual Exam: Controlled Evaluation of Multimodal Agents and World Models},author={{XGEN Labs}},journal={XGEN Labs Technical Report},year={2026},month=sep,note={As core contributor},}
GUI grounding requires precise identification of interface elements from high-resolution screenshots, but processing dense visual inputs with large vision-language models incurs substantial computational overhead. Existing visual token compression methods typically allocate a fixed budget, ignoring the large variability of visual redundancy across different GUI layouts. We show that a fixed visual budget is suboptimal, as the necessity of high-resolution visual evidence varies substantially across interface contexts. We propose FoveaUI, a dynamic high-resolution budget allocation framework for GUI grounding. The agent first reasons over a low-resolution global view and selectively requests high-resolution crops only when fine-grained evidence is required. FoveaUI introduces three coordinated components that progressively address when to seek high-resolution detail, where to acquire it, and how to leverage it effectively: a balanced training strategy for on-demand resolution allocation, a mix-tool strategy paired with a smooth process-level reward for region and granularity selection, and locality-prioritized attention masking for efficient utilization of acquired high-resolution information. Experiments on GUI grounding benchmarks show that our method maintains strong grounding performance while substantially reducing high-resolution visual token costs.
@inproceedings{chen2026foveaui,title={FoveaUI: Foveated On-demand Visual Evidence Acquisition for GUI Grounding},author={Chen, Jingfan and Wang, Lin and Li, Qing},booktitle={Proceedings of the 34th ACM International Conference on Multimedia},year={2026},}
Process supervision improves perceptual grounding in RLVR-trained vision-language models, but it also incurs repeated verifier queries and reshapes the policy by suppressing exploratory entropy. We show that its benefit and cost are empirically separable: grounding gains emerge early, whereas the exploration cost accumulates with continued supervision, holding the policy’s entropy below that of outcome-only RLVR. This hidden behavioral cost shows up as skill tradeoffs on static benchmarks and, more importantly, as a degraded initialization for subsequent outcome-only RLVR that prematurely limits its reachable ceiling. Motivated by this asymmetry, we propose warmup-and-merge: early-stop the process-supervised warmup, merge only a low-rank component of the warmup task vector into the base model, and continue with outcome-only RLVR under an adaptive on-policy reward that targets the model’s current perceptual failures. Across multiple benchmarks and backbones, our method improves perception while preserving exploration, achieving stronger final performance at a fraction of the verifier cost of continuous process supervision.
@inproceedings{chen2026warmup,title={Warm Up with a Rubric, Then Merge: Decoupling Grounding Gains from Reasoning Costs in Vision-Language Models},author={Chen, Jingfan and Wang, Lin and Su, Jingran and Li, Qing},booktitle={Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},year={2026},}
A generalist GUI agent pre-trained on about 20.6 million GUI understanding tasks to strengthen element grounding, then fine-tuned on a unified dataset that maps heterogeneous action formats into one shared action space, with strong results across GUI agent benchmarks on multiple platforms.
@inproceedings{li2025uipro,title={UIPro: Unleashing Superior Interaction Capability For GUI Agents},author={Li, Hongxin and Su, Jingran and Chen, Jingfan and Ju, Zheng and Chen, Yuntao and Li, Qing and Zhang, Zhaoxiang},booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},pages={1613--1623},year={2025},}
An automatic pipeline that uses LLMs to annotate what UI elements do by observing how the interface changes after simulated interactions, with LLM-based rejection and verification to filter noisy labels. The resulting AutoGUI-704k dataset improves GUI grounding of vision-language models and scales with data size.
@inproceedings{li2025autogui,title={AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs},author={Li, Hongxin and Chen, Jingfan and Su, Jingran and Chen, Yuntao and Li, Qing and Zhang, Zhaoxiang},booktitle={Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},year={2025},}
A training-free decoding method that identifies hallucination-related directions in the hidden states of large vision-language models from a small calibration set and steers generation away from them with contrastive decoding.
@inproceedings{su2025activation,title={Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention},author={Su, Jingran and Chen, Jingfan and Li, Hongxin and Chen, Yuntao and Li, Qing and Zhang, Zhaoxiang},booktitle={Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},pages={12964--12974},year={2025},doi={10.18653/v1/2025.acl-long.634},}
@article{wu2023dataset,title={Dataset Condensation for Recommendation},author={Wu, Jiahao and Fan, Wenqi and Chen, Jingfan and Liu, Shengcai and Liu, Qijiong and He, Rui and Li, Qing and Tang, Ke},journal={arXiv preprint arXiv:2310.01038},year={2023},}
@article{chen2023diffusepast,title={DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation},author={Chen, Jingfan and Wang, Yuxi and Wang, Pengfei and Chen, Xiao and Zhang, Zhaoxiang and Lei, Zhen and Li, Qing},journal={arXiv preprint arXiv:2308.01127},year={2023},}
@inproceedings{chen2023fairly,title={Fairly Adaptive Negative Sampling for Recommendations},author={Chen, Xiao and Fan, Wenqi and Chen, Jingfan and Liu, Haochen and Liu, Zitao and Zhang, Zhaoxiang and Li, Qing},booktitle={Proceedings of the ACM Web Conference 2023},pages={3723--3733},year={2023},doi={10.1145/3543507.3583355},}
@inproceedings{chen2022heterophily,title={Towards Self-supervised Learning on Graphs with Heterophily},author={Chen, Jingfan and Zhu, Guanghui and Qi, Yifan and Yuan, Chunfeng and Huang, Yihua},booktitle={Proceedings of the 31st ACM International Conference on Information and Knowledge Management},year={2022},doi={10.1145/3511808.3557478},}
@inproceedings{wu2022disentangled,title={Disentangled Contrastive Learning for Social Recommendation},author={Wu, Jiahao and Fan, Wenqi and Chen, Jingfan and Liu, Shengcai and Li, Qing and Tang, Ke},booktitle={Proceedings of the 31st ACM International Conference on Information and Knowledge Management},year={2022},doi={10.1145/3511808.3557583},}
KGAttack, a black-box attack on recommender systems that uses knowledge graphs to enhance the state representation and candidate selection of a hierarchical reinforcement-learning attacker, which generates fake user profiles under a limited query budget.
@inproceedings{chen2022knowledge,title={Knowledge-enhanced Black-box Attacks for Recommendations},author={Chen, Jingfan and Fan, Wenqi and Zhu, Guanghui and Zhao, Xiangyu and Yuan, Chunfeng and Li, Qing and Huang, Yihua},booktitle={Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining},year={2022},doi={10.1145/3534678.3539359},}
@inproceedings{chen2022autogsr,title={AutoGSR: Neural Architecture Search for Graph-based Session Recommendation},author={Chen, Jingfan and Zhu, Guanghui and Hou, Haojun and Yuan, Chunfeng and Huang, Yihua},booktitle={Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval},year={2022},doi={10.1145/3477495.3531940},}
@article{chen2020semi,title={Semi-supervised Embedding Learning for High-dimensional Bayesian Optimization},author={Chen, Jingfan and Zhu, Guanghui and Yuan, Chunfeng and Huang, Yihua},journal={arXiv preprint arXiv:2005.14601},year={2020},}
