Papers

Field

Top tools here

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Long-horizon task execution and recovery1 paper raises it · 0 address it
Agents struggle with preserving progress, recovering from failures, and verifying physical states over long sequences.
- Has GPT-6-Astra Achieved Embodied Intelligence?raises
Efficient spatial reasoning for real-time interaction2 papers raise it · 2 address it
Current models use slow autoregressive decoding for compact output spaces, limiting interaction speed.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Cross-shot consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across non-consecutive video shots is difficult.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Precise grounding for physical intelligence1 paper raises it · 1 addresses it
General-purpose backbones fail to precisely ground objects in clutter or for tiny targets at control speeds.
- GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitivesraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

video understanding · video language · video generation301 papers49 this week
visual question answering · vision language models · vision language165 papers45 this week
autoencoders · imagenet · autoencoder224 papers40 this week
semantic segmentation · vision language · attention202 papers31 this week
video generation · efficient video · autoregressive video181 papers31 this week
visual reasoning · multimodal reasoning · visual question answering177 papers30 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.CVOct 16
VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings
Ruizhi Chen, Zining Fan, Yudi Guo, Jiang Liu, Zhiyu Lin +3
Fine-grained alignment between text and video object trajectories is a core challenge in multimodal understanding. Existing embedding models excel at global retrieval, yet their single global representation entangles multiple objects and their spatiotemporal dynamics, limiting direct object-level grounding. To address this issue, we propose VTrajEmbed, an embedding model that decomposes a video into object trajectories and aligns each trajectory embedding with text for object-level grounding through retrieval. Our main contributions: (1)Slow-Fast Trajectory Encoding, a dual-branch architecture that establishes trajectory representations by densely sampling low-resolution frames for high-frequency motion cues and sparsely selecting high-resolution frames for scene context, aggregated by a shared LLM backbone; (2)Mask-Guided Contrastive Learning, a training objective built upon these representations that couples an intra-group loss and a batch-level bidirectional loss under a relation mask, excluding unverified relations and preventing confirmed matches from becoming false negatives; (3)Text2Traj-Bench, a trajectory-level text-video benchmark with model-assisted construction and…
PDF arXiv
arXiv cs.AIOct 16
World Action Critic (WAC) Model: Joint Foresight and Criticism for Policy Learning
Xiangteng Zhang, Haoyuan Liu, Yang Guan, Jingliang Duan, Yao Lyu +5
World models have shown considerable promise in sequential decision-making by anticipating the consequences of actions and providing predictive foresight for policy learning. However, learning to reason about counterfactual actions remains challenging, since predictive objectives may underemphasize information critical for action evaluation, while unexecuted actions lack observed futures for direct supervision. To address these challenges, we propose the World Action Critic (WAC) Model, which proposes multimodal action candidates and jointly learns candidate-conditioned future prediction and consequence evaluation. In our proposed framework, critic supervision directly shapes intermediate predictive representations during rollout, encouraging them to retain information needed to distinguish action quality. Observed futures supervise prediction for executed actions, while external task-quality targets provide evaluation signals for counterfactual candidates. Through joint optimization, critic signals from these candidates propagate into predictive representations without requiring their observed futures. Our theoretical analysis characterizes critic-guided feature retention and…
PDF arXiv
arXiv cs.CVOct 16
VISTA: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users
Noé Mosca, Mohamed Rayan Barhdadi, Muhammad Hamza, Joshua Yeh, Suryodaya Bikram Shahi +9
Blindness and low vision limit many people's ability to complete daily tasks, motivating AI assistants that provide useful guidance. Yet, existing benchmarks do not adequately test whether vision models can provide effective everyday assistance. In particular, i) existing benchmarks primarily rely on visual and textual inputs, with limited support for other modalities that are increasingly available in egocentric wearable devices such as gaze, inertial measurements, spatial trajectories, and others; and ii) existing evaluations emphasize scene understanding and question answering, with limited coverage of goal-conditioned guidance for blind and low-vision users across everyday tasks. To bridge this gap, we introduce VISTA (Visually Impaired Scene and Task Annotations), a multimodal egocentric benchmark for goal-oriented assistance for blind and low-vision users. VISTA contains $\sim$1,000 egocentric recordings captured with Project Aria glasses; each recording provides synchronized sensor streams, six of which serve as model inputs, and three different types of annotation, covering ten task categories (TC), from object localization to hazard detection. We evaluate 14…
PDF arXiv
arXiv cs.ROOct 16
CLARO: Clutter Aware Robot Orchestrator for Open-Vocabulary Grasping using VLM Agents
Sungsu Kim, Vineet Bhat, Prashanth Krishnamurthy, Ramesh Karri, Farshad Khorrami
Robots that follow natural-language instructions unlock a wide range of autonomous applications in households, warehouses, and labs. Real-world environments are diverse and often heavily cluttered, requiring robots to plan over multiple steps, act, and replan as they work toward a goal. We present CLARO (CLutter-Aware Robot Orchestrator), a VLM agent that orchestrates a set of perception and manipulation skills: open-vocabulary object detection, segmentation, 6-DoF grasping, and non-prehensile pushing to locate and retrieve a language-specified target in clutter. CLARO maintains a memory of its past actions to support long-horizon execution, and emits manipulation targets (grasp poses/3D waypoints) allowing transfer across robot arms without task-specific training. We evaluate CLARO against a range of methods on a dynamic simulation environment and standard image-based benchmarks, demonstrating strong generalization across object classes, lighting, and clutter levels. CLARO attains an average success rate of 89.1%, a relative improvement of over 16% on the strongest competing method in high cluttered scenes. We further analyze performance along grounding and grasping axes to…
PDF arXiv
arXiv cs.CVOct 16
STRIDE: Evaluating Spatiotemporal Reasoning in Driving Edge Cases
Lieqi Liu, Rui gao, Jia-Chen Gu, Wenbo Hu, Zhaobin Mo +4
In autonomous driving, perceiving surrounding objects is not enough: failing to understand how they move and interact over time can lead to dangerous decisions. Evaluating vision-language models (VLMs) therefore requires testing whether they connect past spatial observations to future consequences. Yet existing benchmarks often assess edge cases through basic perception questions or test temporal understanding in simple scenarios, leaving complex spatiotemporal reasoning underexplored. We introduce STRIDE (SpatioTemporal Reasoning In Driving Edge Cases), an object-centric benchmark for long-tail, safety-critical driving scenarios. STRIDE covers six task families: spatial perception, spatial understanding, temporal memory, temporal extrapolation, trajectory prediction, and scene-context awareness. It pairs multiple-choice questions probing individual spatiotemporal judgments with related free-form visual question answering (VQA) requiring detailed scene explanations and justifications of driving decisions. We prioritize scenes with unusual object behaviors and complex traffic interactions, generate reference answers from structured dataset annotations through automated procedures,…
PDF arXiv
arXiv cs.ROOct 16
Shape-Aware Neural Time Fields for Rigid-Body Motion and Manipulation Planning
Yanming Zhou, Yuchen Liu, Ahmed H. Qureshi
Rigid-body motion planning is fundamental to robotic transport and manipulation, where both object position and orientation determine whether a body can move through cluttered environments. This coupling makes planning chal- lenging because translation and rotation can be constrained very differently by surrounding geometry. We propose Shape- Aware Neural Time Fields, a physics-informed neural motion planning framework that learns a reusable travel-time, or cost- to-go, representation for rigid-body motion while explicitly accounting for these distinct geometric restrictions. The learned field captures local motion difficulty in translation and rotation and is paired with a sampling-based planner that balances progress toward the goal with the cost of reaching nearby poses. We evaluate the framework across 2D and 3D rigid- body planning tasks and non-prehensile manipulation through object pushing in diverse environments. Results show substantial improvements in planning success and inference efficiency over physics-informed neural planning baselines, while remaining competitive with classical sampling-based planners at significantly lower planning times. Ablation studies further…
PDF arXiv

Papers

Topics in this field

video understanding · video language · video generation301 papers49 this week
visual question answering · vision language models · vision language165 papers45 this week
autoencoders · imagenet · autoencoder224 papers40 this week
semantic segmentation · vision language · attention202 papers31 this week
video generation · efficient video · autoregressive video181 papers31 this week
visual reasoning · multimodal reasoning · visual question answering177 papers30 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.CVOct 16
VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings
Ruizhi Chen, Zining Fan, Yudi Guo, Jiang Liu, Zhiyu Lin +3
Fine-grained alignment between text and video object trajectories is a core challenge in multimodal understanding. Existing embedding models excel at global retrieval, yet their single global representation entangles multiple objects and their spatiotemporal dynamics, limiting direct object-level grounding. To address this issue, we propose VTrajEmbed, an embedding model that decomposes a video into object trajectories and aligns each trajectory embedding with text for object-level grounding through retrieval. Our main contributions: (1)Slow-Fast Trajectory Encoding, a dual-branch architecture that establishes trajectory representations by densely sampling low-resolution frames for high-frequency motion cues and sparsely selecting high-resolution frames for scene context, aggregated by a shared LLM backbone; (2)Mask-Guided Contrastive Learning, a training objective built upon these representations that couples an intra-group loss and a batch-level bidirectional loss under a relation mask, excluding unverified relations and preventing confirmed matches from becoming false negatives; (3)Text2Traj-Bench, a trajectory-level text-video benchmark with model-assisted construction and…
PDF arXiv
arXiv cs.AIOct 16
World Action Critic (WAC) Model: Joint Foresight and Criticism for Policy Learning
Xiangteng Zhang, Haoyuan Liu, Yang Guan, Jingliang Duan, Yao Lyu +5
World models have shown considerable promise in sequential decision-making by anticipating the consequences of actions and providing predictive foresight for policy learning. However, learning to reason about counterfactual actions remains challenging, since predictive objectives may underemphasize information critical for action evaluation, while unexecuted actions lack observed futures for direct supervision. To address these challenges, we propose the World Action Critic (WAC) Model, which proposes multimodal action candidates and jointly learns candidate-conditioned future prediction and consequence evaluation. In our proposed framework, critic supervision directly shapes intermediate predictive representations during rollout, encouraging them to retain information needed to distinguish action quality. Observed futures supervise prediction for executed actions, while external task-quality targets provide evaluation signals for counterfactual candidates. Through joint optimization, critic signals from these candidates propagate into predictive representations without requiring their observed futures. Our theoretical analysis characterizes critic-guided feature retention and…
PDF arXiv
arXiv cs.CVOct 16
VISTA: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users
Noé Mosca, Mohamed Rayan Barhdadi, Muhammad Hamza, Joshua Yeh, Suryodaya Bikram Shahi +9
Blindness and low vision limit many people's ability to complete daily tasks, motivating AI assistants that provide useful guidance. Yet, existing benchmarks do not adequately test whether vision models can provide effective everyday assistance. In particular, i) existing benchmarks primarily rely on visual and textual inputs, with limited support for other modalities that are increasingly available in egocentric wearable devices such as gaze, inertial measurements, spatial trajectories, and others; and ii) existing evaluations emphasize scene understanding and question answering, with limited coverage of goal-conditioned guidance for blind and low-vision users across everyday tasks. To bridge this gap, we introduce VISTA (Visually Impaired Scene and Task Annotations), a multimodal egocentric benchmark for goal-oriented assistance for blind and low-vision users. VISTA contains $\sim$1,000 egocentric recordings captured with Project Aria glasses; each recording provides synchronized sensor streams, six of which serve as model inputs, and three different types of annotation, covering ten task categories (TC), from object localization to hazard detection. We evaluate 14…
PDF arXiv
arXiv cs.ROOct 16
CLARO: Clutter Aware Robot Orchestrator for Open-Vocabulary Grasping using VLM Agents
Sungsu Kim, Vineet Bhat, Prashanth Krishnamurthy, Ramesh Karri, Farshad Khorrami
Robots that follow natural-language instructions unlock a wide range of autonomous applications in households, warehouses, and labs. Real-world environments are diverse and often heavily cluttered, requiring robots to plan over multiple steps, act, and replan as they work toward a goal. We present CLARO (CLutter-Aware Robot Orchestrator), a VLM agent that orchestrates a set of perception and manipulation skills: open-vocabulary object detection, segmentation, 6-DoF grasping, and non-prehensile pushing to locate and retrieve a language-specified target in clutter. CLARO maintains a memory of its past actions to support long-horizon execution, and emits manipulation targets (grasp poses/3D waypoints) allowing transfer across robot arms without task-specific training. We evaluate CLARO against a range of methods on a dynamic simulation environment and standard image-based benchmarks, demonstrating strong generalization across object classes, lighting, and clutter levels. CLARO attains an average success rate of 89.1%, a relative improvement of over 16% on the strongest competing method in high cluttered scenes. We further analyze performance along grounding and grasping axes to…
PDF arXiv
arXiv cs.CVOct 16
STRIDE: Evaluating Spatiotemporal Reasoning in Driving Edge Cases
Lieqi Liu, Rui gao, Jia-Chen Gu, Wenbo Hu, Zhaobin Mo +4
In autonomous driving, perceiving surrounding objects is not enough: failing to understand how they move and interact over time can lead to dangerous decisions. Evaluating vision-language models (VLMs) therefore requires testing whether they connect past spatial observations to future consequences. Yet existing benchmarks often assess edge cases through basic perception questions or test temporal understanding in simple scenarios, leaving complex spatiotemporal reasoning underexplored. We introduce STRIDE (SpatioTemporal Reasoning In Driving Edge Cases), an object-centric benchmark for long-tail, safety-critical driving scenarios. STRIDE covers six task families: spatial perception, spatial understanding, temporal memory, temporal extrapolation, trajectory prediction, and scene-context awareness. It pairs multiple-choice questions probing individual spatiotemporal judgments with related free-form visual question answering (VQA) requiring detailed scene explanations and justifications of driving decisions. We prioritize scenes with unusual object behaviors and complex traffic interactions, generate reference answers from structured dataset annotations through automated procedures,…
PDF arXiv
arXiv cs.ROOct 16
Shape-Aware Neural Time Fields for Rigid-Body Motion and Manipulation Planning
Yanming Zhou, Yuchen Liu, Ahmed H. Qureshi
Rigid-body motion planning is fundamental to robotic transport and manipulation, where both object position and orientation determine whether a body can move through cluttered environments. This coupling makes planning chal- lenging because translation and rotation can be constrained very differently by surrounding geometry. We propose Shape- Aware Neural Time Fields, a physics-informed neural motion planning framework that learns a reusable travel-time, or cost- to-go, representation for rigid-body motion while explicitly accounting for these distinct geometric restrictions. The learned field captures local motion difficulty in translation and rotation and is paired with a sampling-based planner that balances progress toward the goal with the cost of reaching nearby poses. We evaluate the framework across 2D and 3D rigid- body planning tasks and non-prehensile manipulation through object pushing in diverse environments. Results show substantial improvements in planning success and inference efficiency over physics-informed neural planning baselines, while remaining competitive with classical sampling-based planners at significantly lower planning times. Ablation studies further…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 4m ago

Multimodal

Models that combine modalities such as vision-language models, video-language and any-to-any models.

518papers this week−29.4% vs last week

All papers · 3.3k With code · 220

Newest Most relevant

3.1k papers this month

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Long-horizon task execution and recovery1 paper raises it · 0 address it
Agents struggle with preserving progress, recovering from failures, and verifying physical states over long sequences.
- Has GPT-6-Astra Achieved Embodied Intelligence?raises
Efficient spatial reasoning for real-time interaction2 papers raise it · 2 address it
Current models use slow autoregressive decoding for compact output spaces, limiting interaction speed.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Cross-shot consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across non-consecutive video shots is difficult.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Precise grounding for physical intelligence1 paper raises it · 1 addresses it
General-purpose backbones fail to precisely ground objects in clutter or for tiny targets at control speeds.
- GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitivesraises · addresses

Named by an AI from these papers' abstracts.