Papers

Field

Top tools here

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Long-horizon task execution and recovery1 paper raises it · 0 address it
Agents struggle with preserving progress, recovering from failures, and verifying physical states over long sequences.
- Has GPT-6-Astra Achieved Embodied Intelligence?raises
Efficient spatial reasoning for real-time interaction2 papers raise it · 2 address it
Current models use slow autoregressive decoding for compact output spaces, limiting interaction speed.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Cross-shot consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across non-consecutive video shots is difficult.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Precise grounding for physical intelligence1 paper raises it · 1 addresses it
General-purpose backbones fail to precisely ground objects in clutter or for tiny targets at control speeds.
- GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitivesraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

video understanding · video language · video generation301 papers50 this week
visual question answering · vision language models · vision language165 papers46 this week
autoencoders · imagenet · autoencoder224 papers44 this week
visual reasoning · multimodal reasoning · visual question answering177 papers33 this week
semantic segmentation · vision language · attention202 papers32 this week
video generation · efficient video · autoregressive video181 papers32 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.CVCodeOct 4▲ 8
Code2Games: Enabling Coding Agents for Gaming World Generation
Wei Wu, Ziyang Xu, Zeyu Zhang, Yang Zhao, Hao Tang
Generating a high-quality gaming world from a natural-language game intent requires joint reasoning about scene structure, spatial layout, gameplay objectives, interactive entities, and executable gameplay logic. Existing coding agents can generate individual assets, scenes, or scripts, but often struggle to maintain consistency across these components. We propose Code2Games, an agentic framework that builds a structured gaming world upon a base Blender world generated from the same game intent. Code2Games coordinates scene analysis, gameplay planning, constrained gaming-world generation, and gaming-engine customization through a shared scene-gameplay representation with persistent element correspondence. After world generation, Code2Games adapts the generated world to Unreal Engine 5 and employs an execution-guided reconstruction process that uses compilation diagnostics, runtime feedback, and gameplay test results to resolve inconsistencies arising during engine adaptation. To systematically evaluate gaming-world generation, we introduce the GameCode4D benchmark, which comprises ten fixed game prompts spanning different levels of scene and gameplay complexity. We evaluate the…
PDF arXiv Code
arXiv cs.AICodeOct 4▲ 2
How corner is a corner case? Percentile control for highway scenario generation
Jiaxi Liu, Hang Zhou, Hangyu Li, Yifan Wang, Keke Long +3
Generating corner-case scenarios with appropriate adversity in a simulation environment is critical for testing an autonomous vehicle (AV) software stack's safety performance before deployment. Existing autonomous-driving scenario generators can enforce specific behavior, adversity, or feasibility conditions, but they provide limited control over how extreme a generated scenario is relative to plausible futures in the same traffic context. This study represents the adversity of a generated scenario as its percentile in the conditional distribution of future risk given the observed history. This view supports calibrated answers to two questions: how "corner" a generated corner-case scenario is and how its "cornerness" can be fine-tuned. To this end, we formulate history-conditioned risk-percentile requests and learn a reference risk distribution that maps each requested percentile to a physical risk target. We then use a percentile-conditioned joint diffusion model with sampling-time risk guidance to generate multi-agent futures, together with a reference-based criterion for evaluating percentile realization. Experiments use the minimum post-encroachment time (PET) between the ego…
PDF arXiv Code
arXiv cs.LGCodeOct 3▲ 4
Learning Discriminative Geometry for Drifting Models
Doudou Zhang, Wenwen Hou, Yilin Chen, Qi Chen
Recently proposed Drifting Models shift iterative distribution refinement from inference to training, enabling effective one-step generation. However, their performance on complex image datasets depends strongly on the representation used to construct the drifting field: pixel-space drifting performs poorly, whereas pretrained feature spaces substantially improve sample quality for reasons that remain unclear. We trace this gap to the discriminative geometry of the representation, which determines sample weighting in kernel density estimation (KDE) and, consequently drift. We introduce persistent representation learning, which continuously learns a more discriminative representation geometry as the generator evolves across batches. We further establish a current-step gradient equivalence between the KDE ratio loss and drift regression loss under matched conditions, connecting density-ratio-based generator optimization to empirical drifting and motivating direct control of the drifting velocity. Across multiple datasets, our method learns effective discriminative representations directly from pixels and reduces FID by approximately $82-95\%$ over the original pixel-space Drifting…
PDF arXiv Code
arXiv cs.ROCodeOct 3▲ 21
PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training
Mingyu Liu, Chonghao Sima, Tianjian Feng, Hanqing Wang, Cong Chen +2
A vision--language--action (VLA) policy can complete complex tasks while ignoring the evidence that should determine its actions. An object held near the wrist camera can displace the instructed target. Language and action show the same pattern: a familiar noun can trigger the operation it was paired with in training even after the verb changes, and a gripper that closed on nothing may lift anyway. We call these dependencies modality shortcuts: regularities in successful demonstrations make visual, lexical, or motor cues sufficient to predict expert actions without the task evidence needed for the underlying decision. More demonstrations of the same kind can raise task success while leaving these shortcuts intact. We propose Perturbot which makes task-relevant evidence easier to use and shortcuts insufficient on their own: it applies task-preserving wrist-view perturbations, enriches instructions with decision-relevant captions, and adds random and failed trajectory segments relabeled with the behavior they contain. It complements scaling by changing what is scaled, and leaves inference unchanged. Moreover, we propose GroundingFscore, an offline score that diagnoses how severely…
PDF arXiv Code
arXiv cs.CVCodeOct 3
Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
Jinzhou Tang, Zijun Zhang, Jing Yang, Yuchen Yan, Kun Zhou +8
Building interactive simulators from real-world observations is a promising way to scale embodied data, but current pipelines still rely heavily on manual environment construction and calibration. We study whether frontier foundation models and coding agents can automate this process end to end. We formulate \emph{autonomous video-to-simulation} as a software engineering task in which an agent observes an embodied video, constructs the corresponding simulated environment and robot behavior, and iteratively refines the result through execution feedback. To evaluate this capability, we introduce \textbf{Video2World}, a benchmark comprising 222 reconstruction instances derived from 189 robot and human demonstration videos. Video2World measures reconstructed worlds along geometric fidelity, dynamic fidelity, and functional correctness, capturing spatial perception, physical reasoning, and executable interaction. Evaluating 9 frontier coding-agent systems reveals a sharp improvement in Task success beginning with Claude Opus 5, rising from below 5\% to over 15\%, while substantial gaps to human-assisted reconstruction remain. We further find that worlds that look better could work…
PDF arXiv Code
arXiv cs.CVCodeOct 3▲ 17
Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency
Sixun Dong, Wei Li, Andong Deng, Qi Qian, Victor Zhu +2
Efficient long-video understanding with vision-language models (VLMs) is often framed as selecting informative frames or visual tokens at a fixed native resolution. We show that per-frame resolution can instead be traded for denser temporal coverage, while front-end decoding latency depends on the size of the candidate pool rather than the final token budget. An empirical study across multiple VLMs and long-video benchmarks yields three findings: dense low-resolution sampling outperforms sparse native-resolution sampling at matched token budgets; resolution-sensitive tasks benefit from selected high-resolution frames; and front-end decoding dominates wall time for hour-long videos. Motivated by these findings, we introduce LoHi, a training-free, single-pass framework that combines a dense low-resolution video stream with sparse high-resolution image frames through the VLM's native video and image pathways. LoHi-Anchor selects high-resolution frames using codec I-frame metadata, while LoHi-SemDiv uses query relevance and visual diversity over CLIP features. Across three long-video benchmarks, LoHi improves average accuracy by 10.6 percentage points over the native-resolution…
PDF arXiv Code

Papers

Topics in this field

video understanding · video language · video generation301 papers50 this week
visual question answering · vision language models · vision language165 papers46 this week
autoencoders · imagenet · autoencoder224 papers44 this week
visual reasoning · multimodal reasoning · visual question answering177 papers33 this week
semantic segmentation · vision language · attention202 papers32 this week
video generation · efficient video · autoregressive video181 papers32 this week

Active contributors · 90 days

Recent papers

All in Research →

arXiv cs.CVCodeOct 4▲ 8
Code2Games: Enabling Coding Agents for Gaming World Generation
Wei Wu, Ziyang Xu, Zeyu Zhang, Yang Zhao, Hao Tang
Generating a high-quality gaming world from a natural-language game intent requires joint reasoning about scene structure, spatial layout, gameplay objectives, interactive entities, and executable gameplay logic. Existing coding agents can generate individual assets, scenes, or scripts, but often struggle to maintain consistency across these components. We propose Code2Games, an agentic framework that builds a structured gaming world upon a base Blender world generated from the same game intent. Code2Games coordinates scene analysis, gameplay planning, constrained gaming-world generation, and gaming-engine customization through a shared scene-gameplay representation with persistent element correspondence. After world generation, Code2Games adapts the generated world to Unreal Engine 5 and employs an execution-guided reconstruction process that uses compilation diagnostics, runtime feedback, and gameplay test results to resolve inconsistencies arising during engine adaptation. To systematically evaluate gaming-world generation, we introduce the GameCode4D benchmark, which comprises ten fixed game prompts spanning different levels of scene and gameplay complexity. We evaluate the…
PDF arXiv Code
arXiv cs.AICodeOct 4▲ 2
How corner is a corner case? Percentile control for highway scenario generation
Jiaxi Liu, Hang Zhou, Hangyu Li, Yifan Wang, Keke Long +3
Generating corner-case scenarios with appropriate adversity in a simulation environment is critical for testing an autonomous vehicle (AV) software stack's safety performance before deployment. Existing autonomous-driving scenario generators can enforce specific behavior, adversity, or feasibility conditions, but they provide limited control over how extreme a generated scenario is relative to plausible futures in the same traffic context. This study represents the adversity of a generated scenario as its percentile in the conditional distribution of future risk given the observed history. This view supports calibrated answers to two questions: how "corner" a generated corner-case scenario is and how its "cornerness" can be fine-tuned. To this end, we formulate history-conditioned risk-percentile requests and learn a reference risk distribution that maps each requested percentile to a physical risk target. We then use a percentile-conditioned joint diffusion model with sampling-time risk guidance to generate multi-agent futures, together with a reference-based criterion for evaluating percentile realization. Experiments use the minimum post-encroachment time (PET) between the ego…
PDF arXiv Code
arXiv cs.LGCodeOct 3▲ 4
Learning Discriminative Geometry for Drifting Models
Doudou Zhang, Wenwen Hou, Yilin Chen, Qi Chen
Recently proposed Drifting Models shift iterative distribution refinement from inference to training, enabling effective one-step generation. However, their performance on complex image datasets depends strongly on the representation used to construct the drifting field: pixel-space drifting performs poorly, whereas pretrained feature spaces substantially improve sample quality for reasons that remain unclear. We trace this gap to the discriminative geometry of the representation, which determines sample weighting in kernel density estimation (KDE) and, consequently drift. We introduce persistent representation learning, which continuously learns a more discriminative representation geometry as the generator evolves across batches. We further establish a current-step gradient equivalence between the KDE ratio loss and drift regression loss under matched conditions, connecting density-ratio-based generator optimization to empirical drifting and motivating direct control of the drifting velocity. Across multiple datasets, our method learns effective discriminative representations directly from pixels and reduces FID by approximately $82-95\%$ over the original pixel-space Drifting…
PDF arXiv Code
arXiv cs.ROCodeOct 3▲ 21
PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training
Mingyu Liu, Chonghao Sima, Tianjian Feng, Hanqing Wang, Cong Chen +2
A vision--language--action (VLA) policy can complete complex tasks while ignoring the evidence that should determine its actions. An object held near the wrist camera can displace the instructed target. Language and action show the same pattern: a familiar noun can trigger the operation it was paired with in training even after the verb changes, and a gripper that closed on nothing may lift anyway. We call these dependencies modality shortcuts: regularities in successful demonstrations make visual, lexical, or motor cues sufficient to predict expert actions without the task evidence needed for the underlying decision. More demonstrations of the same kind can raise task success while leaving these shortcuts intact. We propose Perturbot which makes task-relevant evidence easier to use and shortcuts insufficient on their own: it applies task-preserving wrist-view perturbations, enriches instructions with decision-relevant captions, and adds random and failed trajectory segments relabeled with the behavior they contain. It complements scaling by changing what is scaled, and leaves inference unchanged. Moreover, we propose GroundingFscore, an offline score that diagnoses how severely…
PDF arXiv Code
arXiv cs.CVCodeOct 3
Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
Jinzhou Tang, Zijun Zhang, Jing Yang, Yuchen Yan, Kun Zhou +8
Building interactive simulators from real-world observations is a promising way to scale embodied data, but current pipelines still rely heavily on manual environment construction and calibration. We study whether frontier foundation models and coding agents can automate this process end to end. We formulate \emph{autonomous video-to-simulation} as a software engineering task in which an agent observes an embodied video, constructs the corresponding simulated environment and robot behavior, and iteratively refines the result through execution feedback. To evaluate this capability, we introduce \textbf{Video2World}, a benchmark comprising 222 reconstruction instances derived from 189 robot and human demonstration videos. Video2World measures reconstructed worlds along geometric fidelity, dynamic fidelity, and functional correctness, capturing spatial perception, physical reasoning, and executable interaction. Evaluating 9 frontier coding-agent systems reveals a sharp improvement in Task success beginning with Claude Opus 5, rising from below 5\% to over 15\%, while substantial gaps to human-assisted reconstruction remain. We further find that worlds that look better could work…
PDF arXiv Code
arXiv cs.CVCodeOct 3▲ 17
Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency
Sixun Dong, Wei Li, Andong Deng, Qi Qian, Victor Zhu +2
Efficient long-video understanding with vision-language models (VLMs) is often framed as selecting informative frames or visual tokens at a fixed native resolution. We show that per-frame resolution can instead be traded for denser temporal coverage, while front-end decoding latency depends on the size of the candidate pool rather than the final token budget. An empirical study across multiple VLMs and long-video benchmarks yields three findings: dense low-resolution sampling outperforms sparse native-resolution sampling at matched token budgets; resolution-sensitive tasks benefit from selected high-resolution frames; and front-end decoding dominates wall time for hour-long videos. Motivated by these findings, we introduce LoHi, a training-free, single-pass framework that combines a dense low-resolution video stream with sparse high-resolution image frames through the VLM's native video and image pathways. LoHi-Anchor selects high-resolution frames using codec I-frame metadata, while LoHi-SemDiv uses query relevance and visual diversity over CLIP features. Across three long-video benchmarks, LoHi improves average accuracy by 10.6 percentage points over the native-resolution…
PDF arXiv Code

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 51m ago

Multimodal

Models that combine modalities such as vision-language models, video-language and any-to-any models.

547papers this week−26.7% vs last week

All papers · 3.3k With code · 218

Newest Most relevant

3.1k papers this month

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Long-horizon task execution and recovery1 paper raises it · 0 address it
Agents struggle with preserving progress, recovering from failures, and verifying physical states over long sequences.
- Has GPT-6-Astra Achieved Embodied Intelligence?raises
Efficient spatial reasoning for real-time interaction2 papers raise it · 2 address it
Current models use slow autoregressive decoding for compact output spaces, limiting interaction speed.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Cross-shot consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across non-consecutive video shots is difficult.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Precise grounding for physical intelligence1 paper raises it · 1 addresses it
General-purpose backbones fail to precisely ground objects in clutter or for tiny targets at control speeds.
- GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitivesraises · addresses

Named by an AI from these papers' abstracts.