Papers

Field

Top tools here

Open problems in Multimodal

Lack of holistic benchmarks for omni-modal video understanding1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of models' video grounding and understanding capabilities.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Limited generality and efficiency in in-context learning for embodied agents1 paper raises it · 0 address it
Current embodied agents struggle with generalizing in-context learning, high execution costs, and long-horizon task preservation.
- Has GPT-6-Astra Achieved Embodied Intelligence?raises
Slow and inefficient decision-making for spatial and temporal video tasks2 papers raise it · 2 address it
Autoregressive decoding and answer parsing in VLMs are inefficient for compact output spaces like spatial relations or temporal intervals.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Poor spatial reasoning and grounding precision in real-world control1 paper raises it · 1 addresses it
General-purpose backbones fail to provide precise grounding for tiny or cluttered objects, which is critical for closed-loop control.
- GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitivesraises · addresses
Gap between 2D perception and 3D spatial reasoning for action1 paper raises it · 0 address it
Models struggle to infer 3D geometry and rotation effects from 2D observations, limiting precise interactive control.
- RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignmentraises

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

video understanding · video language · video generation291 papers47 this week
autoencoders · imagenet · autoencoder220 papers40 this week
visual question answering · vision language models · vision language151 papers40 this week
video generation · efficient video · autoregressive video178 papers35 this week
visual reasoning · multimodal reasoning · visual question answering166 papers32 this week
semantic segmentation · vision language · attention196 papers31 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.CVOct 19
PICA-WAM: Visual Perturbation Invariance via Cross-Appearance Self-Alignment in World Action Models
Yude Zou, Yifei Yao, Guanbo Jiao, Jianming Ma, Guanjie Zheng +1
World Action Models (WAMs) jointly predict future video and generate actions, yet remain sensitive to task-irrelevant visual perturbations, i.e., changes in visual appearance that should not affect robot behavior. We identify a gap in conventional WAM training: video prediction encourages representations to preserve appearance information, while action generation does not explicitly enforce invariance to task-irrelevant appearance. Data augmentation exposes WAMs to such perturbations but does not require perturbed and nominal observations to produce consistent action representations, and alignment with external pretrained features does not guarantee the desired invariance. To address this gap, we propose PICA-WAM, a self-supervised training framework that casts perturbation invariance as an auxiliary cross-appearance self-alignment task for WAMs. An exponential-moving-average (EMA) teacher processes the nominal observation to provide a deeper-layer target, while the online WAM processes a perturbed observation and aligns its shallower-layer action representation to this target through a lightweight projection head. Across four strong WAMs evaluated on RoboTwin 2.0, LIBERO-Plus,…
PDF arXiv
arXiv cs.CVCodeOct 19▲ 12
State World Models: Execute the Rules, Generate the Pixels
Chan Yan Ki, Yanran Zhang, Wenzhao Zheng, Xiaofeng Wang, Zheng Zhu +3
Existing world models can generate action-conditioned video from a single image. Many focus on camera movement, allowing users to explore the generated scene. However, navigation alone does not ensure that an action follows persistent rules when its outcome depends on state that is invisible in the current frame. We present Stateful Harness, a framework that separates world evolution from video rendering to support stateful and rule-governed interaction. It maintains an evolving world state under specified rules, renders each resulting event with a frozen video model, and uses the generated clip to update the world state for the next turn. At each turn, the harness resolves the action selected by the user and compiles the resulting event into an instruction, while a preview clip from an editable 3D scene provides spatial and camera guidance. To evaluate rule compliance, we build WorldState-Bench, a benchmark of 50 counterfactual twin pairs that share an initial frame and action but differ in hidden state and required outcome. Under rule-only prompting, the best of the 11 evaluated video backends produces the required endings for both cases in 32% of pairs. For MiniMax-H3,…
PDF arXiv Code
arXiv cs.CVOct 19
Mise-en-Video: Fine-Grained 3D Control for Multi-Shot Video Generation
Yingshuang Zou, Xiaoyu Shi, Yawen Luo, Ziang Li, Qinghe Wang +6
Recent advances in text-to-video generation have enabled high-fidelity single-shot synthesis, yet practical storytelling creation requires multi-shot videos with both strong inter-shot consistency and fine-grained controllability. Existing multi-shot video generation approaches achieve strong consistency but still fall short in controllability. In this paper, we propose Mise-en-Video (MeV), a highly controllable multi-shot video generation framework that allows video creators to conduct staging the multi-shot in the 3D space and then generate the corresponding multi-shot video. To achieve this, our framework combines a hierarchical attention mechanism for multi-level narrative consistency with 3D-aware layout guidance for precise, per-shot spatial control. To support training and evaluation, we develop standardized annotation pipelines and construct a 160K-video dataset with hierarchical multi-shot texts and oriented 3D bounding boxes. Extensive experiments demonstrate that MeV consistently outperforms strong baselines in video quality, inter-shot consistency, and subject-level controllability.
PDF arXiv
arXiv cs.CVOct 19
V-JEPA Hand: Simple, Accurate and Fast 3D Hand and Camera Trajectory Recovery from Egocentric Videos
Ruicheng Li, Qixiu Li, Yu Deng, Sicheng Xu, Jiaolong Yang +1
Egocentric video supports diverse applications, including robot learning, augmented reality, and human activity understanding. Recovering metric 3D hand trajectories from such video requires estimating both hand motion and camera egomotion to express the hands in a consistent reference frame. Many existing pipelines recover hand and camera motion in separate stages, combining hand detection, hand reconstruction, camera tracking, and trajectory refinement. We present V-JEPA Hand, a simple feed-forward video model that jointly reconstructs bilateral hand motion and camera motion from egocentric clips. It uses a self-supervised V-JEPA 2.1 encoder to extract spatio-temporal features from full-frame video and a query-based predictor to decode these features into per-frame hand poses, hand presence, and camera motion. We use the predicted camera motion to express the hand trajectories in a common world frame. The model accepts known focal lengths or predicts them when unavailable. It reconstructs each clip in a single forward pass. We train both components end to end in a single stage on a mixture of diverse pseudo-labelled videos and high-precision 3D annotations. A single model…
PDF arXiv
arXiv cs.CVOct 19
Beyond Omni: A Unified MLLM for Embodied Sensory Understanding
Shiyun Xiong, Dongming Wu, Peiwen Sun, Bokang Yang, Jiaxu Wang +3
Recent embodied systems increasingly build on multimodal large language models (MLLMs) for their pretrained understanding and reasoning capabilities, but their perception of the physical world remains largely limited to basic modalities such as vision and audio. However, broadening MLLMs to more diverse sensory modalities from the physical environment is challenging: their representations are often weakly connected to the MLLM and vary substantially in structure and statistics, while their interpretation still requires shared high-level understanding and reasoning. We therefore propose Unisense, a unified embodied MLLM framework that supports diverse physical sensory modalities within a shared multimodal architecture. Unisense follows a two-stage training process: (i) aligning sensory representations with language to establish a language-accessible interface; and (ii) tuning modality-specific experts within the shared LLM to provide specialized processing for different sensory modalities. To support broad sensory learning, we curate Unisense-1M, a large-scale embodied sensory training corpus comprising 1M samples across 8 sensory modalities. We further evaluate Unisense on 12…
PDF arXiv
arXiv cs.CVOct 19
CheckDistill: Adaptive Checklist Distillation for Efficient Distributional Reward Modeling in Image Editing
Yaqi Cai, Zuan Gao, Mengting Chen, Wei Li, Bo Zheng
Reinforcement learning for instruction-guided image editing requires reward models that provide reliable, accurate, and efficient feedback. Direct scalar prediction compresses multidimensional editing preferences, whereas generative evaluators can expose their judgment through explicit reasoning. However, broad scoring criteria do not specify all instance-dependent editing and preservation constraints, and generating long reasoning trajectories is costly for online optimization. We introduce CheckDistill, a pointwise reward modeling framework that internalizes instruction-adaptive checklists. We decompose each editing condition into atomic, independently verifiable requirements and construct four-dimensional scoring labels together with structured verification evidence. After supervised score initialization, a privileged teacher conditions on checklists, item-level judgments, and dimension-level rationales to supervise the student's own scoring trajectories. A reliability-weighted on-policy self-distillation objective adjusts supervision at score tokens according to agreement between the teacher's expected scores and reference labels, while retaining candidate probability weights…
PDF arXiv

Papers

Topics in this field

video understanding · video language · video generation291 papers47 this week
autoencoders · imagenet · autoencoder220 papers40 this week
visual question answering · vision language models · vision language151 papers40 this week
video generation · efficient video · autoregressive video178 papers35 this week
visual reasoning · multimodal reasoning · visual question answering166 papers32 this week
semantic segmentation · vision language · attention196 papers31 this week

Active contributors · 90 days

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 45m ago

Multimodal

Models that combine modalities such as vision-language models, video-language and any-to-any models.

504papers this week−33.0% vs last week

All papers · 3.2k With code · 200

Newest Most relevant

3k papers this month

Recent papers

All in Research →

arXiv cs.CVOct 19
PICA-WAM: Visual Perturbation Invariance via Cross-Appearance Self-Alignment in World Action Models
Yude Zou, Yifei Yao, Guanbo Jiao, Jianming Ma, Guanjie Zheng +1
World Action Models (WAMs) jointly predict future video and generate actions, yet remain sensitive to task-irrelevant visual perturbations, i.e., changes in visual appearance that should not affect robot behavior. We identify a gap in conventional WAM training: video prediction encourages representations to preserve appearance information, while action generation does not explicitly enforce invariance to task-irrelevant appearance. Data augmentation exposes WAMs to such perturbations but does not require perturbed and nominal observations to produce consistent action representations, and alignment with external pretrained features does not guarantee the desired invariance. To address this gap, we propose PICA-WAM, a self-supervised training framework that casts perturbation invariance as an auxiliary cross-appearance self-alignment task for WAMs. An exponential-moving-average (EMA) teacher processes the nominal observation to provide a deeper-layer target, while the online WAM processes a perturbed observation and aligns its shallower-layer action representation to this target through a lightweight projection head. Across four strong WAMs evaluated on RoboTwin 2.0, LIBERO-Plus,…
PDF arXiv
arXiv cs.CVCodeOct 19▲ 12
State World Models: Execute the Rules, Generate the Pixels
Chan Yan Ki, Yanran Zhang, Wenzhao Zheng, Xiaofeng Wang, Zheng Zhu +3
Existing world models can generate action-conditioned video from a single image. Many focus on camera movement, allowing users to explore the generated scene. However, navigation alone does not ensure that an action follows persistent rules when its outcome depends on state that is invisible in the current frame. We present Stateful Harness, a framework that separates world evolution from video rendering to support stateful and rule-governed interaction. It maintains an evolving world state under specified rules, renders each resulting event with a frozen video model, and uses the generated clip to update the world state for the next turn. At each turn, the harness resolves the action selected by the user and compiles the resulting event into an instruction, while a preview clip from an editable 3D scene provides spatial and camera guidance. To evaluate rule compliance, we build WorldState-Bench, a benchmark of 50 counterfactual twin pairs that share an initial frame and action but differ in hidden state and required outcome. Under rule-only prompting, the best of the 11 evaluated video backends produces the required endings for both cases in 32% of pairs. For MiniMax-H3,…
PDF arXiv Code
arXiv cs.CVOct 19
Mise-en-Video: Fine-Grained 3D Control for Multi-Shot Video Generation
Yingshuang Zou, Xiaoyu Shi, Yawen Luo, Ziang Li, Qinghe Wang +6
Recent advances in text-to-video generation have enabled high-fidelity single-shot synthesis, yet practical storytelling creation requires multi-shot videos with both strong inter-shot consistency and fine-grained controllability. Existing multi-shot video generation approaches achieve strong consistency but still fall short in controllability. In this paper, we propose Mise-en-Video (MeV), a highly controllable multi-shot video generation framework that allows video creators to conduct staging the multi-shot in the 3D space and then generate the corresponding multi-shot video. To achieve this, our framework combines a hierarchical attention mechanism for multi-level narrative consistency with 3D-aware layout guidance for precise, per-shot spatial control. To support training and evaluation, we develop standardized annotation pipelines and construct a 160K-video dataset with hierarchical multi-shot texts and oriented 3D bounding boxes. Extensive experiments demonstrate that MeV consistently outperforms strong baselines in video quality, inter-shot consistency, and subject-level controllability.
PDF arXiv
arXiv cs.CVOct 19
V-JEPA Hand: Simple, Accurate and Fast 3D Hand and Camera Trajectory Recovery from Egocentric Videos
Ruicheng Li, Qixiu Li, Yu Deng, Sicheng Xu, Jiaolong Yang +1
Egocentric video supports diverse applications, including robot learning, augmented reality, and human activity understanding. Recovering metric 3D hand trajectories from such video requires estimating both hand motion and camera egomotion to express the hands in a consistent reference frame. Many existing pipelines recover hand and camera motion in separate stages, combining hand detection, hand reconstruction, camera tracking, and trajectory refinement. We present V-JEPA Hand, a simple feed-forward video model that jointly reconstructs bilateral hand motion and camera motion from egocentric clips. It uses a self-supervised V-JEPA 2.1 encoder to extract spatio-temporal features from full-frame video and a query-based predictor to decode these features into per-frame hand poses, hand presence, and camera motion. We use the predicted camera motion to express the hand trajectories in a common world frame. The model accepts known focal lengths or predicts them when unavailable. It reconstructs each clip in a single forward pass. We train both components end to end in a single stage on a mixture of diverse pseudo-labelled videos and high-precision 3D annotations. A single model…
PDF arXiv
arXiv cs.CVOct 19
Beyond Omni: A Unified MLLM for Embodied Sensory Understanding
Shiyun Xiong, Dongming Wu, Peiwen Sun, Bokang Yang, Jiaxu Wang +3
Recent embodied systems increasingly build on multimodal large language models (MLLMs) for their pretrained understanding and reasoning capabilities, but their perception of the physical world remains largely limited to basic modalities such as vision and audio. However, broadening MLLMs to more diverse sensory modalities from the physical environment is challenging: their representations are often weakly connected to the MLLM and vary substantially in structure and statistics, while their interpretation still requires shared high-level understanding and reasoning. We therefore propose Unisense, a unified embodied MLLM framework that supports diverse physical sensory modalities within a shared multimodal architecture. Unisense follows a two-stage training process: (i) aligning sensory representations with language to establish a language-accessible interface; and (ii) tuning modality-specific experts within the shared LLM to provide specialized processing for different sensory modalities. To support broad sensory learning, we curate Unisense-1M, a large-scale embodied sensory training corpus comprising 1M samples across 8 sensory modalities. We further evaluate Unisense on 12…
PDF arXiv
arXiv cs.CVOct 19
CheckDistill: Adaptive Checklist Distillation for Efficient Distributional Reward Modeling in Image Editing
Yaqi Cai, Zuan Gao, Mengting Chen, Wei Li, Bo Zheng
Reinforcement learning for instruction-guided image editing requires reward models that provide reliable, accurate, and efficient feedback. Direct scalar prediction compresses multidimensional editing preferences, whereas generative evaluators can expose their judgment through explicit reasoning. However, broad scoring criteria do not specify all instance-dependent editing and preservation constraints, and generating long reasoning trajectories is costly for online optimization. We introduce CheckDistill, a pointwise reward modeling framework that internalizes instruction-adaptive checklists. We decompose each editing condition into atomic, independently verifiable requirements and construct four-dimensional scoring labels together with structured verification evidence. After supervised score initialization, a privileged teacher conditions on checklists, item-level judgments, and dimension-level rationales to supervise the student's own scoring trajectories. A reliability-weighted on-policy self-distillation objective adjusts supervision at score tokens according to agreement between the teacher's expected scores and reference labels, while retaining candidate probability weights…
PDF arXiv

Open problems in Multimodal

Lack of holistic benchmarks for omni-modal video understanding1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of models' video grounding and understanding capabilities.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Limited generality and efficiency in in-context learning for embodied agents1 paper raises it · 0 address it
Current embodied agents struggle with generalizing in-context learning, high execution costs, and long-horizon task preservation.
- Has GPT-6-Astra Achieved Embodied Intelligence?raises
Slow and inefficient decision-making for spatial and temporal video tasks2 papers raise it · 2 address it
Autoregressive decoding and answer parsing in VLMs are inefficient for compact output spaces like spatial relations or temporal intervals.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Poor spatial reasoning and grounding precision in real-world control1 paper raises it · 1 addresses it
General-purpose backbones fail to provide precise grounding for tiny or cluttered objects, which is critical for closed-loop control.
- GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitivesraises · addresses
Gap between 2D perception and 3D spatial reasoning for action1 paper raises it · 0 address it
Models struggle to infer 3D geometry and rotation effects from 2D observations, limiting precise interactive control.
- RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignmentraises

Named by an AI from these papers' abstracts.