Papers

Field

Top tools here

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Slow and inefficient spatial reasoning in multimodal models2 papers raise it · 2 address it
Autoregressive decoding and answer parsing are costly for compact output spaces like spatial relations or coordinates.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Poor long-horizon consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across nonconsecutive video shots is difficult without shared references.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Limited fine-grained control and interaction in 3D spatial reasoning1 paper raises it · 0 address it
Models struggle with precise 3D-to-2D projection alignment and sensitive interaction history in spatial tasks.
- RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignmentraises
Scalable and realistic training environments for embodied agents1 paper raises it · 1 addresses it
Building faithful and realistic interactive worlds for agent training remains a bottleneck.
- AgentGarten: Code Worlds for Evolving Agentsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

video understanding · video language · video generation309 papers54 this week
autoencoders · imagenet · autoencoder233 papers45 this week
visual question answering · vision language models · vision language170 papers45 this week
video generation · efficient video · autoregressive video189 papers35 this week
semantic segmentation · vision language · attention208 papers34 this week
reasoning vision language · spatial reasoning · reasoning vision160 papers30 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.CVOct 13
Bifocal: Training-Free Mixed Granularity Visual Token Pruning for Multimodal LLMs
Peiran Yin, Hao Wu, Zihao Yang, Fengyu Zhou, Yijuan Liang +4
Visual token pruning for multimodal large language models (MLLMs) appears highly effective, with existing methods retaining over 95% of unpruned performance on common VQA benchmarks using a quarter of the visual tokens or fewer. However, our extensive evaluation of ten methods across three MLLM families and five ability groups reveals a critical gap: strong general performance often comes at the cost of severe degradation on fine-grained tasks such as OCR and visual grounding. We trace these failures to two limitations of the prevailing paradigm of retaining the highest-ranked native patches. A single importance ranking overlooks the complementary evidence required by different tasks, while sparse patch selection disrupts both image content and spatial integrity. To address these limitations, we propose Bifocal, a training-free, two-stage pruning method built on a mixed-granularity framework. It allocates separate budgets to query-focused and image-intrinsic evidence, complemented by a pooled coarse representation of the entire image. Before the LLM, Bifocal matches question words to visual regions through a soft conjunction balanced against norm-debiased image saliency, while…
PDF arXiv
arXiv cs.CVOct 13
ReDiS: Reasoning-Diagnosis Self-Evolution for Multimodal Reasoning in Remote Sensing
Yi Liu, Wentao Jiang, Shuo Ni, Zhiming Luo, Di Wang +3
Multimodal large language models (MLLMs) have shown strong potential for remote sensing understanding, yet reliable reasoning remains challenging when tasks require fine-grained visual grounding and multi-step inference. Existing approaches typically rely on either costly pre-constructed reasoning trajectories or self-generated signals such as response consistency, which can reinforce shared errors and provide unreliable supervision. We introduce \textbf{Reasoning--Diagnosis Self-Evolution (ReDiS)}, a framework that separates reasoning exploration from candidate evaluation while allowing the two processes to evolve jointly. ReDiS instantiates two role-specific experts from a shared multimodal backbone: a \textbf{Reasoner} that explores diverse visual evidence--answer candidates and a \textbf{Diagnoser} that jointly evaluates the complete candidate set and provides credibility-aware feedback. As the Reasoner evolves, it exposes new candidate distributions and failure modes; the Diagnoser adapts to these changes and, in turn, reshapes the learning signal for subsequent exploration. To handle groups in which all sampled answers are incorrect and group-relative advantages vanish, we…
PDF arXiv
arXiv cs.CVOct 13
Hallucination-aware Faithful Reasoning in Remote Sensing
Yi Liu, Jing Zhang, Xiang Feng, Xiaoyu Tian, Haonan Guo +3
Multimodal large language models (MLLMs) have demonstrated promising capabilities for remote sensing reasoning, yet they frequently generate visually unsupported reasoning that contradicts the visual evidence. Existing post-training methods optimize primarily for final-answer correctness, providing limited supervision for identifying and correcting hallucinations that arise during the reasoning process. To address this limitation, we propose \textbf{FaithRS}, a hallucination-aware self-reflection framework for faithful reasoning in remote sensing. We first construct \textbf{SFHD-43K}, a dataset of 43K structured, fine-grained hallucination annotations, and train a detector to identify hallucination types, underlying causes, and supporting visual evidence. FaithRS then incorporates this diagnostic feedback into a Safe Detector-Guided Corrective Rollout framework, where a safety-aware gate selectively determines whether and how to revise the response. Training is driven by a composite reward that jointly optimizes answer accuracy, reasoning faithfulness, hallucination reduction, format compliance, and revision safety. Experiments with Qwen3.5 backbones (0.8B, 4B, and 9B) show…
PDF arXiv
arXiv cs.CVCodeOct 13▲ 130
Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes
Yu Wang, Rui Dai, Wei Chen, Yi Wang, Bo Dang +3
City-scale 3D grounding is commonly formulated as candidate ranking, which requires precomputed instance proposals and becomes inefficient in large outdoor scenes. We propose Urbex, an agentic vision-language framework that instead casts grounding as active spatial search. Urbex represents a city-scale 3D scene as an interactive multi-view environment, allowing a VLM agent to query landmarks, zoom into local regions, render oblique views, and commit a grounded bounding box. We optimize this tool-use policy with reinforcement learning on CityRefer, using smooth localization rewards and lightweight shaping rewards for efficient exploration. Experiments show that Urbex achieves the best Hit@1 and IoU on CityRefer, supports candidate-free grounding without ground-truth boxes, and is substantially faster than the strongest city-scale ranking baseline. Zero-shot results on STPLS3D-Refer further indicate improved cross-scene robustness.
PDF arXiv Code
arXiv cs.AIOct 13
ABot-ER: Unified Embodied Reasoning through Agentic Data-Model Co-Evolution
Qiuyu Ding, Heng-Da Xu, Changda Xia, Jiajun Shen, Zihao Guan +12
We introduce ABot Embodied Reasoning (ABot-ER), a unified embodied reasoning foundation model that breaks the tension between breadth and depth in embodied reasoning. Embodied intelligence demands the joint mastery of perception, planning, and execution, yet reconciling generality with specialization across this spectrum remains an open challenge. Within a single unified model, ABot-ER commands specialist-level depth across all three competencies, spanning spatial, scene, and temporal reasoning in perception; long-horizon action organization in planning; and point grounding and affordance in execution. The Agentic Data-Model Self-Evolution Engine continuously adapts data strategies to the model's evolving capability state. The Task-Conditioned Adaptive Training Strategy uses task tokens to activate lightweight low-rank residuals over a shared backbone, separating updates from heterogeneous output formats while retaining shared embodied knowledge. Across a systematic evaluation on twelve benchmarks covering three core competencies and five capability dimensions, ABot-ER achieves state-of-the-art overall performance across perception, planning, and execution within a unified…
PDF arXiv
arXiv cs.ROCodeOct 13▲ 0
Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation
Youpeng Wen, Gang Wang, Tianyu Huang, Weiwei Zhang, Zhijian Huang +3
A robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awareness of task progress and determine when to advance. To address these challenges, we introduce Orchestra, an agent harness that separates task planning and execution management from action generation. Orchestra decomposes a complete multimodal instruction into ordered subtasks, tracks completed steps, and updates the locations of relevant objects and destinations during execution. It maintains a shared context containing the active subtask, spatial grounding, and recent observations. An action policy uses this context to generate actions, while a separately trained progress evaluator estimates when to advance to the next subtask. To evaluate whether robots trained on atomic instructions and short trajectories can execute longer subtask sequences, we introduce a benchmark named RMBench-Compose. Orchestra achieves a mean success rate of $68.22\%$ across nine RMBench tasks, outperforming Mem-0 by $26.22$ percentage points. On RMBench-Compose, it achieves an Average Success Length (ASL) of $7.35$ subtasks, $2.46$ times that of…
PDF arXiv Code

Papers

Topics in this field

video understanding · video language · video generation309 papers54 this week
autoencoders · imagenet · autoencoder233 papers45 this week
visual question answering · vision language models · vision language170 papers45 this week
video generation · efficient video · autoregressive video189 papers35 this week
semantic segmentation · vision language · attention208 papers34 this week
reasoning vision language · spatial reasoning · reasoning vision160 papers30 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.CVOct 13
Bifocal: Training-Free Mixed Granularity Visual Token Pruning for Multimodal LLMs
Peiran Yin, Hao Wu, Zihao Yang, Fengyu Zhou, Yijuan Liang +4
Visual token pruning for multimodal large language models (MLLMs) appears highly effective, with existing methods retaining over 95% of unpruned performance on common VQA benchmarks using a quarter of the visual tokens or fewer. However, our extensive evaluation of ten methods across three MLLM families and five ability groups reveals a critical gap: strong general performance often comes at the cost of severe degradation on fine-grained tasks such as OCR and visual grounding. We trace these failures to two limitations of the prevailing paradigm of retaining the highest-ranked native patches. A single importance ranking overlooks the complementary evidence required by different tasks, while sparse patch selection disrupts both image content and spatial integrity. To address these limitations, we propose Bifocal, a training-free, two-stage pruning method built on a mixed-granularity framework. It allocates separate budgets to query-focused and image-intrinsic evidence, complemented by a pooled coarse representation of the entire image. Before the LLM, Bifocal matches question words to visual regions through a soft conjunction balanced against norm-debiased image saliency, while…
PDF arXiv
arXiv cs.CVOct 13
ReDiS: Reasoning-Diagnosis Self-Evolution for Multimodal Reasoning in Remote Sensing
Yi Liu, Wentao Jiang, Shuo Ni, Zhiming Luo, Di Wang +3
Multimodal large language models (MLLMs) have shown strong potential for remote sensing understanding, yet reliable reasoning remains challenging when tasks require fine-grained visual grounding and multi-step inference. Existing approaches typically rely on either costly pre-constructed reasoning trajectories or self-generated signals such as response consistency, which can reinforce shared errors and provide unreliable supervision. We introduce \textbf{Reasoning--Diagnosis Self-Evolution (ReDiS)}, a framework that separates reasoning exploration from candidate evaluation while allowing the two processes to evolve jointly. ReDiS instantiates two role-specific experts from a shared multimodal backbone: a \textbf{Reasoner} that explores diverse visual evidence--answer candidates and a \textbf{Diagnoser} that jointly evaluates the complete candidate set and provides credibility-aware feedback. As the Reasoner evolves, it exposes new candidate distributions and failure modes; the Diagnoser adapts to these changes and, in turn, reshapes the learning signal for subsequent exploration. To handle groups in which all sampled answers are incorrect and group-relative advantages vanish, we…
PDF arXiv
arXiv cs.CVOct 13
Hallucination-aware Faithful Reasoning in Remote Sensing
Yi Liu, Jing Zhang, Xiang Feng, Xiaoyu Tian, Haonan Guo +3
Multimodal large language models (MLLMs) have demonstrated promising capabilities for remote sensing reasoning, yet they frequently generate visually unsupported reasoning that contradicts the visual evidence. Existing post-training methods optimize primarily for final-answer correctness, providing limited supervision for identifying and correcting hallucinations that arise during the reasoning process. To address this limitation, we propose \textbf{FaithRS}, a hallucination-aware self-reflection framework for faithful reasoning in remote sensing. We first construct \textbf{SFHD-43K}, a dataset of 43K structured, fine-grained hallucination annotations, and train a detector to identify hallucination types, underlying causes, and supporting visual evidence. FaithRS then incorporates this diagnostic feedback into a Safe Detector-Guided Corrective Rollout framework, where a safety-aware gate selectively determines whether and how to revise the response. Training is driven by a composite reward that jointly optimizes answer accuracy, reasoning faithfulness, hallucination reduction, format compliance, and revision safety. Experiments with Qwen3.5 backbones (0.8B, 4B, and 9B) show…
PDF arXiv
arXiv cs.CVCodeOct 13▲ 130
Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes
Yu Wang, Rui Dai, Wei Chen, Yi Wang, Bo Dang +3
City-scale 3D grounding is commonly formulated as candidate ranking, which requires precomputed instance proposals and becomes inefficient in large outdoor scenes. We propose Urbex, an agentic vision-language framework that instead casts grounding as active spatial search. Urbex represents a city-scale 3D scene as an interactive multi-view environment, allowing a VLM agent to query landmarks, zoom into local regions, render oblique views, and commit a grounded bounding box. We optimize this tool-use policy with reinforcement learning on CityRefer, using smooth localization rewards and lightweight shaping rewards for efficient exploration. Experiments show that Urbex achieves the best Hit@1 and IoU on CityRefer, supports candidate-free grounding without ground-truth boxes, and is substantially faster than the strongest city-scale ranking baseline. Zero-shot results on STPLS3D-Refer further indicate improved cross-scene robustness.
PDF arXiv Code
arXiv cs.AIOct 13
ABot-ER: Unified Embodied Reasoning through Agentic Data-Model Co-Evolution
Qiuyu Ding, Heng-Da Xu, Changda Xia, Jiajun Shen, Zihao Guan +12
We introduce ABot Embodied Reasoning (ABot-ER), a unified embodied reasoning foundation model that breaks the tension between breadth and depth in embodied reasoning. Embodied intelligence demands the joint mastery of perception, planning, and execution, yet reconciling generality with specialization across this spectrum remains an open challenge. Within a single unified model, ABot-ER commands specialist-level depth across all three competencies, spanning spatial, scene, and temporal reasoning in perception; long-horizon action organization in planning; and point grounding and affordance in execution. The Agentic Data-Model Self-Evolution Engine continuously adapts data strategies to the model's evolving capability state. The Task-Conditioned Adaptive Training Strategy uses task tokens to activate lightweight low-rank residuals over a shared backbone, separating updates from heterogeneous output formats while retaining shared embodied knowledge. Across a systematic evaluation on twelve benchmarks covering three core competencies and five capability dimensions, ABot-ER achieves state-of-the-art overall performance across perception, planning, and execution within a unified…
PDF arXiv
arXiv cs.ROCodeOct 13▲ 0
Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation
Youpeng Wen, Gang Wang, Tianyu Huang, Weiwei Zhang, Zhijian Huang +3
A robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awareness of task progress and determine when to advance. To address these challenges, we introduce Orchestra, an agent harness that separates task planning and execution management from action generation. Orchestra decomposes a complete multimodal instruction into ordered subtasks, tracks completed steps, and updates the locations of relevant objects and destinations during execution. It maintains a shared context containing the active subtask, spatial grounding, and recent observations. An action policy uses this context to generate actions, while a separately trained progress evaluator estimates when to advance to the next subtask. To evaluate whether robots trained on atomic instructions and short trajectories can execute longer subtask sequences, we introduce a benchmark named RMBench-Compose. Orchestra achieves a mean success rate of $68.22\%$ across nine RMBench tasks, outperforming Mem-0 by $26.22$ percentage points. On RMBench-Compose, it achieves an Average Success Length (ASL) of $7.35$ subtasks, $2.46$ times that of…
PDF arXiv Code

Activity · 12 weeks

Fields · AI · updated 9m ago

Multimodal

Models that combine modalities such as vision-language models, video-language and any-to-any models.

560papers this week−22.4% vs last week

All papers · 3.4k With code · 230

Newest Most relevant

3.1k papers this month

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Slow and inefficient spatial reasoning in multimodal models2 papers raise it · 2 address it
Autoregressive decoding and answer parsing are costly for compact output spaces like spatial relations or coordinates.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Poor long-horizon consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across nonconsecutive video shots is difficult without shared references.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Limited fine-grained control and interaction in 3D spatial reasoning1 paper raises it · 0 address it
Models struggle with precise 3D-to-2D projection alignment and sensitive interaction history in spatial tasks.
- RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignmentraises
Scalable and realistic training environments for embodied agents1 paper raises it · 1 addresses it
Building faithful and realistic interactive worlds for agent training remains a bottleneck.
- AgentGarten: Code Worlds for Evolving Agentsraises · addresses

Named by an AI from these papers' abstracts.