Papers

Field

Top tools here

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Long-horizon task execution and recovery1 paper raises it · 0 address it
Agents struggle with preserving progress, recovering from failures, and verifying physical states over long sequences.
- Has GPT-6-Astra Achieved Embodied Intelligence?raises
Efficient spatial reasoning for real-time interaction2 papers raise it · 2 address it
Current models use slow autoregressive decoding for compact output spaces, limiting interaction speed.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Cross-shot consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across non-consecutive video shots is difficult.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Precise grounding for physical intelligence1 paper raises it · 1 addresses it
General-purpose backbones fail to precisely ground objects in clutter or for tiny targets at control speeds.
- GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitivesraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

video understanding · video language · video generation301 papers50 this week
visual question answering · vision language models · vision language165 papers46 this week
autoencoders · imagenet · autoencoder224 papers44 this week
visual reasoning · multimodal reasoning · visual question answering177 papers34 this week
video generation · efficient video · autoregressive video181 papers33 this week
semantic segmentation · vision language · attention202 papers32 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.CVOct 18
PLANE: Plug-and-Play Fine-Grained Visual Experts for Hallucination Mitigation in Large Vision-Language Models
Jianghao Yin, Lang Yu, Qianxi Yu, Qin Chen, Jie Zhou +2
Large Vision-Language Models (LVLMs) commonly suffer from hallucinations, which appear as fabricating non-existent objects or outputting incorrect attributes and spatial relationships in images. Previous works empirically indicate that CLIP-style vision encoders tend to focus on primary objects while ignoring image details, which is one of the causes of hallucinations. To address this issue, we propose plug-and-play fine-grained visual experts (PLANE) comprising two components: LoRA visual experts operating on CLIP for multi-aspect image encoding, and an instruction-aware gate network for adaptive image feature fusion. During training, we construct a fine-grained image-caption pairs dataset, where images are described from three aspects: objects, attributes, and relations between objects. Image-caption pairs from different aspects are used to train the three fine-grained LoRA experts respectively. Aiming at comprehensively evaluate our method's capabilities, we manually annotate a fine-grained hallucination evaluation benchmark for LVLMs which is GPT4-independent during evaluation phase (GIFT). We conduct experiments both on popular hallucination evaluation benchmarks and GIFT.…
PDF arXiv
arXiv cs.CVOct 18
Reflection-Guided Dynamics Rectification for Text-to-Image Generation
Shuo Xu, Hanrui Ying, Debin Liu, Jianjun Li, Zhiyuan Ma
Fine-grained rectification of existing generations under a limited additional budget remains a key problem in test-time optimization (TTO) for image generation. As for prior approaches, they could identify residual errors, but could not directly ensure that the model acts on the feedback during reasoning. Such errors stem from multiple sources, such as the initial noise or text prompt, and ultimately induce an execution gap of failing to handle potential deviations along the trajectory generating. However, classic models generally cannot remedy this gap. To narrow it, we propose Reflection-Guided Dynamics Rectification (RGDR), which turns reflections on completed outputs into corrective interventions on the process of sampling dynamics at the trajectory level. Specifically, RGDR could impact every sampling step by adapting to instantaneous intervention. Moreover, RGDR generates flexible candidates while adjudicating between them and the reference. As experiments show, rectification from RGDR starts from the existing base trajectory and reflection, and RGDR is substantially more effective than former trajectory exploration. Under matched budgets for additional image generation,…
PDF arXiv
arXiv cs.ROOct 18
Compile Once, Run Cheap: Frozen, Portable Skills for Embodied Agents, and What Recursive Self-Improvement Adds
Jingwen Bai, Guochao Bai, Canbo Zhang, Shuangqi Li, Minchao Hu +2
Embodied AI lacks the data-rich pretraining that gave language models their agents and skills, so new manipulation capability costs demonstrations per task or a frontier model in every episode. We study a third route: a strong general model acts only as an offline author that writes, runs, diagnoses and repairs a manipulation skill in simulation until it passes an exam on layouts drawn after freezing; the frozen skill then runs as code alone or under a cheap caller that reads only its description. We call this skill-level recursive self-improvement Skill-RSI and specify a compile protocol (isolated, audited author; hidden validation layouts; no-feedback and inherit-only controls; tamper-evident ledger) and a reuse protocol that runs the identical files with executors from another model family. On 7 RoboTwin 2.0 tasks with cluttered tables, Claude Code compiled 4 tasks into 7 frozen skills, 5 from a first draft needing only qualification, at USD 4.29-8.22 of author cost each (USD 55 per skill counting the 13 failed campaigns). The other three tasks did not compile in 18 campaigns, including five with GPT-6 and a pre-registered 150-layout run: on the hardest task every campaign…
PDF arXiv
arXiv cs.CVOct 18▲ 19
Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation
Junwei Wen, Deshui Miao, Guangming Lu, Ming-Hsuan Yang, Xin Li +1
Video Reasoning Segmentation (VRS) requires a model to identify an implicitly described target and predict its pixel-level mask trajectory, while resolving the spatial relations and temporal events that determine the answer. Text-only Chain-of-Thought (CoT) can explain a target choice without making the underlying visual references explicit, leaving a \emph{video reference gap} between linguistic inference and the evidence distributed across frames. We introduce \textbf{Temporal Chain of Video Primitives (TCoVP)}, a spatiotemporally grounded form of Video CoT that organizes query-relevant observations into a chronological reasoning chain. Each primitive associates an entity reference with temporal support and timestamped spatial evidence, and the chain interleaves these references with language to guide reasoning about which entity is relevant, when the queried event holds, and where its evidence appears. The selected target primitives then provide box prompts and temporal support for video mask prediction. We construct TCoVP-SFT-15k and TCoVP-RL-13k to learn this representation through supervised fine-tuning and chain-aligned reinforcement learning. Extensive experiments across…
PDF arXiv
arXiv cs.CVOct 18
Sufficient Frame Subsets for Video Understanding
Lahiru Wijayasingha
Understanding when during a video a model makes its critical observations is essential for interpreting video-understanding systems. However, existing attribution methods often spread evidence diffusely across near-duplicate frames. We introduce Sufficient Frame Subsets: compact sets of frames in which all other frames are masked, yet each subset independently preserves the model's prediction. We evaluate our method on two complementary benchmarks, UCF101 and Something-Something v2, using both CNN- and transformer-based architectures. Our results show that the proposed algorithm identifies highly compact and effective frame subsets for explaining model decisions. We also find that models capable of relying on a greater number of sufficient frame subsets tend to perform better, possibly because they are more robust, consistent with prior research. Finally, the extracted frame subsets help us analyze model failure cases, reveal which aspects of a video the model has learned to emphasize, and identify opportunities for improving future training procedures.
PDF arXiv
arXiv cs.CVOct 18
MoCKEFA: Model-Conditioned Keyframe Extraction and Fair Attribution for Video Model Explanation
Lahiru Wijayasingha
We show that a video can be represented with just a few keyframes without sacrificing a given model's belief about the video. Our method is the first of its kind that extracts these keyframes conditioned on the model's prediction, which is massively helpful for building explainable video models. We then fairly distribute the importance of the video to the model among the keyframes with game theoretically fair Shapley values. Through qualitative and quantitative experiments, we show that keyframe attribution is superior to the traditional frame attribution. Further, we conduct preliminary exploration on highlighting the important changes among the frames by looking at the differences between adjacent keyframes.
PDF arXiv

Papers

Topics in this field

video understanding · video language · video generation301 papers50 this week
visual question answering · vision language models · vision language165 papers46 this week
autoencoders · imagenet · autoencoder224 papers44 this week
visual reasoning · multimodal reasoning · visual question answering177 papers34 this week
video generation · efficient video · autoregressive video181 papers33 this week
semantic segmentation · vision language · attention202 papers32 this week

Active contributors · 90 days

Activity · 12 weeks

Coming with more sources: repos

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Long-horizon task execution and recovery1 paper raises it · 0 address it
Agents struggle with preserving progress, recovering from failures, and verifying physical states over long sequences.
- Has GPT-6-Astra Achieved Embodied Intelligence?raises
Efficient spatial reasoning for real-time interaction2 papers raise it · 2 address it
Current models use slow autoregressive decoding for compact output spaces, limiting interaction speed.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Cross-shot consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across non-consecutive video shots is difficult.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Precise grounding for physical intelligence1 paper raises it · 1 addresses it
General-purpose backbones fail to precisely ground objects in clutter or for tiny targets at control speeds.
- GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitivesraises · addresses

Named by an AI from these papers' abstracts.

Fields · AI · updated 45m ago

Multimodal

Models that combine modalities such as vision-language models, video-language and any-to-any models.

551papers this week−26.5% vs last week

All papers · 3.3k With code · 218

Newest Most relevant

3.1k papers this month

Recent papers

All in Research →

arXiv cs.CVOct 18
PLANE: Plug-and-Play Fine-Grained Visual Experts for Hallucination Mitigation in Large Vision-Language Models
Jianghao Yin, Lang Yu, Qianxi Yu, Qin Chen, Jie Zhou +2
Large Vision-Language Models (LVLMs) commonly suffer from hallucinations, which appear as fabricating non-existent objects or outputting incorrect attributes and spatial relationships in images. Previous works empirically indicate that CLIP-style vision encoders tend to focus on primary objects while ignoring image details, which is one of the causes of hallucinations. To address this issue, we propose plug-and-play fine-grained visual experts (PLANE) comprising two components: LoRA visual experts operating on CLIP for multi-aspect image encoding, and an instruction-aware gate network for adaptive image feature fusion. During training, we construct a fine-grained image-caption pairs dataset, where images are described from three aspects: objects, attributes, and relations between objects. Image-caption pairs from different aspects are used to train the three fine-grained LoRA experts respectively. Aiming at comprehensively evaluate our method's capabilities, we manually annotate a fine-grained hallucination evaluation benchmark for LVLMs which is GPT4-independent during evaluation phase (GIFT). We conduct experiments both on popular hallucination evaluation benchmarks and GIFT.…
PDF arXiv
arXiv cs.CVOct 18
Reflection-Guided Dynamics Rectification for Text-to-Image Generation
Shuo Xu, Hanrui Ying, Debin Liu, Jianjun Li, Zhiyuan Ma
Fine-grained rectification of existing generations under a limited additional budget remains a key problem in test-time optimization (TTO) for image generation. As for prior approaches, they could identify residual errors, but could not directly ensure that the model acts on the feedback during reasoning. Such errors stem from multiple sources, such as the initial noise or text prompt, and ultimately induce an execution gap of failing to handle potential deviations along the trajectory generating. However, classic models generally cannot remedy this gap. To narrow it, we propose Reflection-Guided Dynamics Rectification (RGDR), which turns reflections on completed outputs into corrective interventions on the process of sampling dynamics at the trajectory level. Specifically, RGDR could impact every sampling step by adapting to instantaneous intervention. Moreover, RGDR generates flexible candidates while adjudicating between them and the reference. As experiments show, rectification from RGDR starts from the existing base trajectory and reflection, and RGDR is substantially more effective than former trajectory exploration. Under matched budgets for additional image generation,…
PDF arXiv
arXiv cs.ROOct 18
Compile Once, Run Cheap: Frozen, Portable Skills for Embodied Agents, and What Recursive Self-Improvement Adds
Jingwen Bai, Guochao Bai, Canbo Zhang, Shuangqi Li, Minchao Hu +2
Embodied AI lacks the data-rich pretraining that gave language models their agents and skills, so new manipulation capability costs demonstrations per task or a frontier model in every episode. We study a third route: a strong general model acts only as an offline author that writes, runs, diagnoses and repairs a manipulation skill in simulation until it passes an exam on layouts drawn after freezing; the frozen skill then runs as code alone or under a cheap caller that reads only its description. We call this skill-level recursive self-improvement Skill-RSI and specify a compile protocol (isolated, audited author; hidden validation layouts; no-feedback and inherit-only controls; tamper-evident ledger) and a reuse protocol that runs the identical files with executors from another model family. On 7 RoboTwin 2.0 tasks with cluttered tables, Claude Code compiled 4 tasks into 7 frozen skills, 5 from a first draft needing only qualification, at USD 4.29-8.22 of author cost each (USD 55 per skill counting the 13 failed campaigns). The other three tasks did not compile in 18 campaigns, including five with GPT-6 and a pre-registered 150-layout run: on the hardest task every campaign…
PDF arXiv
arXiv cs.CVOct 18▲ 19
Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation
Junwei Wen, Deshui Miao, Guangming Lu, Ming-Hsuan Yang, Xin Li +1
Video Reasoning Segmentation (VRS) requires a model to identify an implicitly described target and predict its pixel-level mask trajectory, while resolving the spatial relations and temporal events that determine the answer. Text-only Chain-of-Thought (CoT) can explain a target choice without making the underlying visual references explicit, leaving a \emph{video reference gap} between linguistic inference and the evidence distributed across frames. We introduce \textbf{Temporal Chain of Video Primitives (TCoVP)}, a spatiotemporally grounded form of Video CoT that organizes query-relevant observations into a chronological reasoning chain. Each primitive associates an entity reference with temporal support and timestamped spatial evidence, and the chain interleaves these references with language to guide reasoning about which entity is relevant, when the queried event holds, and where its evidence appears. The selected target primitives then provide box prompts and temporal support for video mask prediction. We construct TCoVP-SFT-15k and TCoVP-RL-13k to learn this representation through supervised fine-tuning and chain-aligned reinforcement learning. Extensive experiments across…
PDF arXiv
arXiv cs.CVOct 18
Sufficient Frame Subsets for Video Understanding
Lahiru Wijayasingha
Understanding when during a video a model makes its critical observations is essential for interpreting video-understanding systems. However, existing attribution methods often spread evidence diffusely across near-duplicate frames. We introduce Sufficient Frame Subsets: compact sets of frames in which all other frames are masked, yet each subset independently preserves the model's prediction. We evaluate our method on two complementary benchmarks, UCF101 and Something-Something v2, using both CNN- and transformer-based architectures. Our results show that the proposed algorithm identifies highly compact and effective frame subsets for explaining model decisions. We also find that models capable of relying on a greater number of sufficient frame subsets tend to perform better, possibly because they are more robust, consistent with prior research. Finally, the extracted frame subsets help us analyze model failure cases, reveal which aspects of a video the model has learned to emphasize, and identify opportunities for improving future training procedures.
PDF arXiv
arXiv cs.CVOct 18
MoCKEFA: Model-Conditioned Keyframe Extraction and Fair Attribution for Video Model Explanation
Lahiru Wijayasingha
We show that a video can be represented with just a few keyframes without sacrificing a given model's belief about the video. Our method is the first of its kind that extracts these keyframes conditioned on the model's prediction, which is massively helpful for building explainable video models. We then fairly distribute the importance of the video to the model among the keyframes with game theoretically fair Shapley values. Through qualitative and quantitative experiments, we show that keyframe attribution is superior to the traditional frame attribution. Further, we conduct preliminary exploration on highlighting the important changes among the frames by looking at the differences between adjacent keyframes.
PDF arXiv