Papers

Field

Top tools here

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Slow and inefficient spatial reasoning in multimodal models2 papers raise it · 2 address it
Autoregressive decoding and answer parsing are costly for compact output spaces like spatial relations or coordinates.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Poor long-horizon consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across nonconsecutive video shots is difficult without shared references.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Limited fine-grained control and interaction in 3D spatial reasoning1 paper raises it · 0 address it
Models struggle with precise 3D-to-2D projection alignment and sensitive interaction history in spatial tasks.
- RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignmentraises
Scalable and realistic training environments for embodied agents1 paper raises it · 1 addresses it
Building faithful and realistic interactive worlds for agent training remains a bottleneck.
- AgentGarten: Code Worlds for Evolving Agentsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

video understanding · video language · video generation309 papers55 this week
autoencoders · imagenet · autoencoder233 papers47 this week
visual question answering · vision language models · vision language170 papers47 this week
video generation · efficient video · autoregressive video189 papers36 this week
semantic segmentation · vision language · attention208 papers34 this week
visual reasoning · multimodal reasoning · visual question answering179 papers32 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.CVOct 13
Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs
Aimon Rahman, Joymallya Chakraborty, Vidit Bansal, Abhishek Dan
Multimodal large language models have advanced rapidly, but most remain optimized for single-image inputs. Multi-image reasoning is costly because each image adds many visual tokens, increasing encoding and attention demands. Existing techniques that address the above issue, including visual token pruning, token merging, visual token compression, learned resampling, hierarchical compression, retrieval-augmented visual reasoning, and memory-based architectures, have largely been designed for single-image scenarios, where redundancy is primarily spatial and local. Multi-image reasoning introduces additional challenges: redundancy may occur across images, task-relevant evidence may be sparse or distributed, and aggressive per-image compression can remove details needed for comparison, counting, temporal reasoning, retrieval, or evidence grounding. Moreover, existing token-reduction methods are often either (i) training-free but inference-inefficient, typically relying on accessing attention matrices to get token-importance that may be impractical to access with memory-efficient attention implementations such as FlashAttention, or (ii) training-dependent, requiring the base MLLM to…
PDF arXiv
arXiv cs.CVOct 13
PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts
Ruixiao Dong, Jianmin Bao, Bozhi Luan, Zijie Meng, Xin Tao +4
Pixel-space diffusion models provide an appealing alternative to latent-space diffusion models by directly modeling raw images without relying on a learned tokenizer. However, scaling pixel-space models has predominantly followed dense strategies, tightly coupling model capacity with active computation. In this work, we show that standard LLM-style Mixture-of-Experts (MoE) architectures can be effectively transferred to pixel-space diffusion. Building on this finding, we introduce PixelMoE, a pixel-space generative framework that sparsely scales model capacity with standard LLM-style MoE, without diffusion-specific routing mechanisms or auxiliary routing objectives. Specifically, PixelMoE combines standard Token-Choice (TC) routing with fine-grained routed experts for input-adaptive sparse computation, a shared expert for capturing common representations, and a bias-based load-balancing strategy. To complement sparse capacity scaling, PixelMoE further introduces Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure formation to fine-grained detail refinement. At comparable active parameter counts, PixelMoE…
PDF arXiv
arXiv cs.CVOct 13
RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark
Mahsa Khoshnoodi, Sarah Adel Bargal
Repetition counting requires identifying and enumerating action cycles over time. While state-of-the-art video language models describe actions fluently, they fail to reliably count how many times an action repeats. To investigate this limitation, we introduce RepProbe, a benchmark of 42K real and synthetic video clips containing exactly k in [1, 20] complete repetitions, with controls for clip position, temporal coverage, and duration. These controls allow us to study the counting failure directly and locate where in the model it arises. Counting accuracy collapses at high counts across the video language models we evaluate, and controlled experiments show this collapse is not explained by frame sampling alone. We compare model outputs with linear probes trained on the final-layer representations used to generate those outputs. We find a consistent representation-to-output dissociation: substantial count information remains linearly decodable even when models fail to report the correct count. Further analysis shows this failure arises primarily at the output stage, where answers concentrate on a small set of preferred numbers rather than the available count information. The…
PDF arXiv
arXiv cs.ROOct 13
LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models
Andrea Yaoyun Cui, Hanyu Wang, Xinyi Wang, Bangzheng Li, Lin Zhang +4
Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted, calibrations drift, controllers run at different rates, and end-effector pads wear down. We present LIBERO-RIG, a benchmark that perturbs only the robot's hardware and setup while holding every LIBERO task fixed. The benchmark spans eleven perturbation axes, including camera mounting and optics, camera-to-arm calibration, actuation gain, control rate, proprioceptive noise, base pose, table height, and gripper pads. Each axis has five severity levels, yielding 210 conditions across the four LIBERO suites. We evaluate nine recent vision-language-action models using 500 episodes per condition and substantiate each finding with rigorous statistical tests. Pooled success falls from 0.932 to 0.767 at the middle severity and 0.509 at the highest severity. Performance degradation varies substantially across perturbation axes and action interfaces, and training on visual perturbations does not mitigate faults that leave camera images unchanged. We release the benchmark together with manifests specifying the fixed perturbation parameters for…
PDF arXiv
arXiv cs.AIOct 13
Who Improves the Robot? Toward Physical Recursive Self-Improvement
Hai Zhou, Fan Yang, Zeyi Chen, Lejun Zhang, Chenyang Fu +25
Robots are increasingly autonomous in task execution, yet their improvement remains largely human-driven. This study investigates Physical Recursive Self-Improvement (Physical RSI), in which physical experience guides lasting updates to both a robot's capabilities and its improvement procedure, making subsequent improvement more effective across tasks. Physical RSI faces challenges from noisy observations, continuous dynamics, costly trials, and irreversible consequences. We present an L0-L5 framework that differentiates capability-level from meta-level updates and within-task from across-task gains, reserving L5 for independently verified recursive gain. We study existing methods around harness adaptation and model training, alongside safety and control mechanisms that govern execution and successive updates. Our analysis reveals that, despite substantial progress in self-improvement, the surveyed literature does not yet demonstrate Physical RSI as defined by our framework. We identify research priorities in reliable feedback, causal attribution, safety preservation, and benchmarking, outlining a path toward robots that autonomously improve how they learn while ensuring safety.…
PDF arXiv
arXiv cs.CVOct 13
Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?
Juntong Feng, Qingan Liu, Jingru Dong, Yue Zhu, Weiming Zheng +6
Vision-Language Models (VLMs) have achieved strong performance in visual perception and spatial reasoning, leading to growing interest in using them as high-level planners for embodied AI. However, physical tasks require models to integrate spatial information, reason over temporal states, and generate multi-step plans under various constraints, raising the question of whether current VLMs can reliably support such reasoning and planning independently of execution. We introduce Plan-Bench, a benchmark for evaluating the high-level reasoning and multi-step planning capabilities of VLMs relevant to embodied worlds, while isolating the influence of action execution. It covers three complementary dimensions, spanning 16 subclasses and 699 curated samples. Extensive evaluation of 28 VLMs reveals room for improvement, with even the strongest model achieving only 75.0% accuracy. We further find that model scaling generally improves these capabilities. Ablation studies also indicate larger models show increasing potential as high-level planners. Overall, Plan-Bench provides a lightweight probe for evaluating VLMs' reasoning and planning capabilities relevant to embodied environments and…
PDF arXiv

Papers

Topics in this field

video understanding · video language · video generation309 papers55 this week
autoencoders · imagenet · autoencoder233 papers47 this week
visual question answering · vision language models · vision language170 papers47 this week
video generation · efficient video · autoregressive video189 papers36 this week
semantic segmentation · vision language · attention208 papers34 this week
visual reasoning · multimodal reasoning · visual question answering179 papers32 this week

Recent papers

All papers in Research

arXiv cs.CVOct 13
Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs
Aimon Rahman, Joymallya Chakraborty, Vidit Bansal, Abhishek Dan
Multimodal large language models have advanced rapidly, but most remain optimized for single-image inputs. Multi-image reasoning is costly because each image adds many visual tokens, increasing encoding and attention demands. Existing techniques that address the above issue, including visual token pruning, token merging, visual token compression, learned resampling, hierarchical compression, retrieval-augmented visual reasoning, and memory-based architectures, have largely been designed for single-image scenarios, where redundancy is primarily spatial and local. Multi-image reasoning introduces additional challenges: redundancy may occur across images, task-relevant evidence may be sparse or distributed, and aggressive per-image compression can remove details needed for comparison, counting, temporal reasoning, retrieval, or evidence grounding. Moreover, existing token-reduction methods are often either (i) training-free but inference-inefficient, typically relying on accessing attention matrices to get token-importance that may be impractical to access with memory-efficient attention implementations such as FlashAttention, or (ii) training-dependent, requiring the base MLLM to…
PDF arXiv
arXiv cs.CVOct 13
PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts
Ruixiao Dong, Jianmin Bao, Bozhi Luan, Zijie Meng, Xin Tao +4
Pixel-space diffusion models provide an appealing alternative to latent-space diffusion models by directly modeling raw images without relying on a learned tokenizer. However, scaling pixel-space models has predominantly followed dense strategies, tightly coupling model capacity with active computation. In this work, we show that standard LLM-style Mixture-of-Experts (MoE) architectures can be effectively transferred to pixel-space diffusion. Building on this finding, we introduce PixelMoE, a pixel-space generative framework that sparsely scales model capacity with standard LLM-style MoE, without diffusion-specific routing mechanisms or auxiliary routing objectives. Specifically, PixelMoE combines standard Token-Choice (TC) routing with fine-grained routed experts for input-adaptive sparse computation, a shared expert for capturing common representations, and a bias-based load-balancing strategy. To complement sparse capacity scaling, PixelMoE further introduces Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure formation to fine-grained detail refinement. At comparable active parameter counts, PixelMoE…
PDF arXiv
arXiv cs.CVOct 13
RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark
Mahsa Khoshnoodi, Sarah Adel Bargal
Repetition counting requires identifying and enumerating action cycles over time. While state-of-the-art video language models describe actions fluently, they fail to reliably count how many times an action repeats. To investigate this limitation, we introduce RepProbe, a benchmark of 42K real and synthetic video clips containing exactly k in [1, 20] complete repetitions, with controls for clip position, temporal coverage, and duration. These controls allow us to study the counting failure directly and locate where in the model it arises. Counting accuracy collapses at high counts across the video language models we evaluate, and controlled experiments show this collapse is not explained by frame sampling alone. We compare model outputs with linear probes trained on the final-layer representations used to generate those outputs. We find a consistent representation-to-output dissociation: substantial count information remains linearly decodable even when models fail to report the correct count. Further analysis shows this failure arises primarily at the output stage, where answers concentrate on a small set of preferred numbers rather than the available count information. The…
PDF arXiv
arXiv cs.ROOct 13
LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models
Andrea Yaoyun Cui, Hanyu Wang, Xinyi Wang, Bangzheng Li, Lin Zhang +4
Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted, calibrations drift, controllers run at different rates, and end-effector pads wear down. We present LIBERO-RIG, a benchmark that perturbs only the robot's hardware and setup while holding every LIBERO task fixed. The benchmark spans eleven perturbation axes, including camera mounting and optics, camera-to-arm calibration, actuation gain, control rate, proprioceptive noise, base pose, table height, and gripper pads. Each axis has five severity levels, yielding 210 conditions across the four LIBERO suites. We evaluate nine recent vision-language-action models using 500 episodes per condition and substantiate each finding with rigorous statistical tests. Pooled success falls from 0.932 to 0.767 at the middle severity and 0.509 at the highest severity. Performance degradation varies substantially across perturbation axes and action interfaces, and training on visual perturbations does not mitigate faults that leave camera images unchanged. We release the benchmark together with manifests specifying the fixed perturbation parameters for…
PDF arXiv
arXiv cs.AIOct 13
Who Improves the Robot? Toward Physical Recursive Self-Improvement
Hai Zhou, Fan Yang, Zeyi Chen, Lejun Zhang, Chenyang Fu +25
Robots are increasingly autonomous in task execution, yet their improvement remains largely human-driven. This study investigates Physical Recursive Self-Improvement (Physical RSI), in which physical experience guides lasting updates to both a robot's capabilities and its improvement procedure, making subsequent improvement more effective across tasks. Physical RSI faces challenges from noisy observations, continuous dynamics, costly trials, and irreversible consequences. We present an L0-L5 framework that differentiates capability-level from meta-level updates and within-task from across-task gains, reserving L5 for independently verified recursive gain. We study existing methods around harness adaptation and model training, alongside safety and control mechanisms that govern execution and successive updates. Our analysis reveals that, despite substantial progress in self-improvement, the surveyed literature does not yet demonstrate Physical RSI as defined by our framework. We identify research priorities in reliable feedback, causal attribution, safety preservation, and benchmarking, outlining a path toward robots that autonomously improve how they learn while ensuring safety.…
PDF arXiv
arXiv cs.CVOct 13
Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?
Juntong Feng, Qingan Liu, Jingru Dong, Yue Zhu, Weiming Zheng +6
Vision-Language Models (VLMs) have achieved strong performance in visual perception and spatial reasoning, leading to growing interest in using them as high-level planners for embodied AI. However, physical tasks require models to integrate spatial information, reason over temporal states, and generate multi-step plans under various constraints, raising the question of whether current VLMs can reliably support such reasoning and planning independently of execution. We introduce Plan-Bench, a benchmark for evaluating the high-level reasoning and multi-step planning capabilities of VLMs relevant to embodied worlds, while isolating the influence of action execution. It covers three complementary dimensions, spanning 16 subclasses and 699 curated samples. Extensive evaluation of 28 VLMs reveals room for improvement, with even the strongest model achieving only 75.0% accuracy. We further find that model scaling generally improves these capabilities. Ablation studies also indicate larger models show increasing potential as high-level planners. Overall, Plan-Bench provides a lightweight probe for evaluating VLMs' reasoning and planning capabilities relevant to embodied environments and…
PDF arXiv

Active contributors · 90 days

Activity · 12 weeks

Fields · AI · updated 38m ago

Multimodal

Models that combine modalities such as vision-language models, video-language and any-to-any models.

576papers this week−20.7% vs last week

All papers · 3.4k With code · 228

Newest Most relevant

3.1k papers this month

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Slow and inefficient spatial reasoning in multimodal models2 papers raise it · 2 address it
Autoregressive decoding and answer parsing are costly for compact output spaces like spatial relations or coordinates.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Poor long-horizon consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across nonconsecutive video shots is difficult without shared references.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Limited fine-grained control and interaction in 3D spatial reasoning1 paper raises it · 0 address it
Models struggle with precise 3D-to-2D projection alignment and sensitive interaction history in spatial tasks.
- RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignmentraises
Scalable and realistic training environments for embodied agents1 paper raises it · 1 addresses it
Building faithful and realistic interactive worlds for agent training remains a bottleneck.
- AgentGarten: Code Worlds for Evolving Agentsraises · addresses

Named by an AI from these papers' abstracts.