Papers

Field

Top tools here

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Slow and inefficient spatial reasoning in multimodal models2 papers raise it · 2 address it
Autoregressive decoding and answer parsing are costly for compact output spaces like spatial relations or coordinates.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Poor long-horizon consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across nonconsecutive video shots is difficult without shared references.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Limited fine-grained control and interaction in 3D spatial reasoning1 paper raises it · 0 address it
Models struggle with precise 3D-to-2D projection alignment and sensitive interaction history in spatial tasks.
- RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignmentraises
Scalable and realistic training environments for embodied agents1 paper raises it · 1 addresses it
Building faithful and realistic interactive worlds for agent training remains a bottleneck.
- AgentGarten: Code Worlds for Evolving Agentsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

video understanding · video language · video generation309 papers56 this week
visual question answering · vision language models · vision language170 papers48 this week
autoencoders · imagenet · autoencoder233 papers47 this week
video generation · efficient video · autoregressive video189 papers38 this week
semantic segmentation · vision language · attention208 papers35 this week
visual reasoning · multimodal reasoning · visual question answering179 papers32 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.CVOct 14
FPV-CRASH: A Dataset for Offline Crash Moment Localization in Drone First-Person-View Videos
Jifan Zhang, Tingyu Wang, Qian Liu, Zhedong Zheng
Crash Moment Localization aims to localize the major physical impact that interrupts controlled flight in an onboard first-person-view (FPV) video recorded by a drone, thereby providing a temporal reference for retrospective crash analysis in aerial robotics. However, existing crash-video datasets predominantly target surveillance or dash-camera footage, and their annotation schemes and task formulations are not designed for temporal localization of the major physical impact in UAV onboard FPV videos. To address this gap, we present FPV-CRASH, a dataset and fixed-protocol reference evaluation for offline Crash Moment Localization in drone FPV videos. FPV-CRASH contains 1,052 drone FPV clips (526 Crash and 526 Normal); each Crash clip has a single-frame reference annotation for the Crash Moment, and every clip is paired with four English captions automatically generated from sampled frames and the known Crash/Normal label. Normal clips carry no timestamp target and provide training-time background supervision. The captions support an auxiliary Text-to-Video Retrieval task, whereas localization inference remains video-only. We further provide a heatmap-based reference model that…
PDF arXiv
arXiv cs.AIOct 14
LabOPBench: A Multimodal Benchmark for Evidence-Grounded Laboratory Decision Making
Jingyuan Li, Yujie Liu, Yijie Dai, Xiaoyu Yuan, Nian Ran +16
Laboratory experiments involve more than following predefined protocols; they require adaptive decision-making as the experimental state changes. Existing scientific benchmarks rarely assess such evidence-based decisions in laboratory settings. Here we present LabOPBench, a multimodal benchmark of 756 expert-authored, independently reviewed questions in organic synthesis and materials science. Each item pairs laboratory evidence with procedural context and an open-ended judgment, evaluated using a question-specific scoring rubric. Eight multimodal models obtain overall normalized scores of 28.8-45.8%, leaving substantial room for improvement. On a controlled 100-question subset, the strongest models outperform undergraduate participants on average but remain below the average performance of PhD participants. Expert analysis of responses from the three leading models identifies Evidence Misreading and Evidence-Decision Conflict as the dominant error patterns. These results reveal that current models still struggle to turn multimodal perception into reliable laboratory decision-making.
PDF arXiv
arXiv cs.CVOct 14
Discriminative Score Function: Turning Pretrained Models into Functional Generative Priors
Junhoo Lee, Hyeonjin Kim, Sangbum Han, Nojun Kwak
Discriminative models are trained with task-specific objectives such as classification, detection, or representation learning, yet their practical utility depends on generalizing beyond the training task to unseen samples from the same domain. This suggests that a discriminative model may retain an implicit trace of the data manifold beyond its task labels. We ask whether this trace can be exposed as a generative signal without task-specific guidance. We introduce the Discriminative Score Function (DSF), which extracts an image-space generative field from the loss-gradient geometry of a pretrained discriminative model. DSF provides a training-free update rule that moves images toward model-consistent visual regions without retraining or task-specific targets. We apply DSF to diverse pretrained discriminative models, including ResNet classifiers, DETR detectors, and DINOv2 ViT encoders. DSF generates unconditional images from noise and supports conditional guidance, editing, inpainting, and explanation through the same field. DSF-generated samples also provide effective zero-shot quantization calibration when real images or labels are unavailable, preserving feature geometry…
PDF arXiv
arXiv cs.CLOct 14
MakSafe: A Spatiotemporal Semantic Framework for Formal Specification and Compliance Checking of Road Vehicle Safety Rules
Maksym Labzhaniia, Julien Deantoni, Marie-Agn{è}s Peraldi-Frati, Fr{é}d{é}ric Mallet
Scenario-based validation is essential for automated road vehicles but cannot cover the full variability of openworld traffic. We present MakSafe, a formal spatiotemporal framework for specifying and checking road vehicle safety rules through semantic zones, action-based forecasting, and trace-tree reasoning, avoiding uniform time discretization. We formalize the road network, vehicle states, and occupancy zones, then build executable rules for front-vehicle identification and timeto-collision (TTC) verification. The evaluation combines a tracedependent case study with a scalability benchmark over traffic density (N ), horizon (H), and branch factor (B). MakSafe yields branch-specific verdicts, deriving safe and unsafe outcomes from the same initial scene, and re-evaluates a scenario after a control update. A single rule check stays well below a millisecond across the reported sweep, and longer horizons slow the TTC check.
PDF arXiv
arXiv cs.LGOct 14
What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization
Julia Barth, Armin Sommer, Kunvar Thaman, Jannik Schilling
Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but does not pursue the intended goal. However, we still have a weak understanding of what happens to the intended goal in misgeneralizing agents. Across three Procgen environments and five robot tasks, we find three distinct forms of policy-goal misalignment: (1) functional target substitution, where behavior follows an alternative goal, (2) a policy shortcut, where behavior follows a learned action strategy, and (3) cue competition between intended and proxy goal cues. We distinguish these regimes by tracing where goal information remains available internally, how strongly it influences the policy, and what role it plays in action selection, and find that the goal is lost at different depths. Where the goal reaches the decision, as in cue competition, steering redirects the policy to the goal's location, raising intended-goal reach from 16.0% to 95.3%. Where it reaches the policy but not the decision, only new downstream wiring recovers goal reach, from 31.6% to 84.3% in the environment with the action shortcut. Where it is inert, neither works. Our results show that goal…
PDF arXiv
arXiv cs.CVCodeOct 14▲ 17
A Diagnostic Study of World-Space Human Recovery Under High Camera Dynamics
Hao-Jen Chien, Wei-Lun Chao, Yu-Lun Liu
Monocular world-space human recovery methods excel on current benchmarks, but these rarely include fast camera motion. We introduce the K-Pop stress test, broadcast dance videos that combine fast, large camera motion with dense multi-person choreography. Leading methods produce plausible per-frame meshes on these videos, yet their world-space trajectories drift far from the performers' stage positions. Through a systematic diagnosis, we show that the camera and human estimates each remain self-consistent, and that the drift appears when they are combined at mismatched scales. Each is recovered at its own scale, so a mismatch between the two can turn camera motion into human drift that grows with camera displacement and becomes severe in broadcast dance videos. We derive a closed-form correction that estimates this mismatch and applies one scale to all performers, removing the drift while keeping their relative positions in one shared world frame. It requires no retraining and works with existing backends. Combined with physical constraints, the correction yields stable world-space reconstruction for choreography modeling, scene editing, data generation, and virtual reshooting.…
PDF arXiv Code

Papers

Topics in this field

video understanding · video language · video generation309 papers56 this week
visual question answering · vision language models · vision language170 papers48 this week
autoencoders · imagenet · autoencoder233 papers47 this week
video generation · efficient video · autoregressive video189 papers38 this week
semantic segmentation · vision language · attention208 papers35 this week
visual reasoning · multimodal reasoning · visual question answering179 papers32 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.CVOct 14
FPV-CRASH: A Dataset for Offline Crash Moment Localization in Drone First-Person-View Videos
Jifan Zhang, Tingyu Wang, Qian Liu, Zhedong Zheng
Crash Moment Localization aims to localize the major physical impact that interrupts controlled flight in an onboard first-person-view (FPV) video recorded by a drone, thereby providing a temporal reference for retrospective crash analysis in aerial robotics. However, existing crash-video datasets predominantly target surveillance or dash-camera footage, and their annotation schemes and task formulations are not designed for temporal localization of the major physical impact in UAV onboard FPV videos. To address this gap, we present FPV-CRASH, a dataset and fixed-protocol reference evaluation for offline Crash Moment Localization in drone FPV videos. FPV-CRASH contains 1,052 drone FPV clips (526 Crash and 526 Normal); each Crash clip has a single-frame reference annotation for the Crash Moment, and every clip is paired with four English captions automatically generated from sampled frames and the known Crash/Normal label. Normal clips carry no timestamp target and provide training-time background supervision. The captions support an auxiliary Text-to-Video Retrieval task, whereas localization inference remains video-only. We further provide a heatmap-based reference model that…
PDF arXiv
arXiv cs.AIOct 14
LabOPBench: A Multimodal Benchmark for Evidence-Grounded Laboratory Decision Making
Jingyuan Li, Yujie Liu, Yijie Dai, Xiaoyu Yuan, Nian Ran +16
Laboratory experiments involve more than following predefined protocols; they require adaptive decision-making as the experimental state changes. Existing scientific benchmarks rarely assess such evidence-based decisions in laboratory settings. Here we present LabOPBench, a multimodal benchmark of 756 expert-authored, independently reviewed questions in organic synthesis and materials science. Each item pairs laboratory evidence with procedural context and an open-ended judgment, evaluated using a question-specific scoring rubric. Eight multimodal models obtain overall normalized scores of 28.8-45.8%, leaving substantial room for improvement. On a controlled 100-question subset, the strongest models outperform undergraduate participants on average but remain below the average performance of PhD participants. Expert analysis of responses from the three leading models identifies Evidence Misreading and Evidence-Decision Conflict as the dominant error patterns. These results reveal that current models still struggle to turn multimodal perception into reliable laboratory decision-making.
PDF arXiv
arXiv cs.CVOct 14
Discriminative Score Function: Turning Pretrained Models into Functional Generative Priors
Junhoo Lee, Hyeonjin Kim, Sangbum Han, Nojun Kwak
Discriminative models are trained with task-specific objectives such as classification, detection, or representation learning, yet their practical utility depends on generalizing beyond the training task to unseen samples from the same domain. This suggests that a discriminative model may retain an implicit trace of the data manifold beyond its task labels. We ask whether this trace can be exposed as a generative signal without task-specific guidance. We introduce the Discriminative Score Function (DSF), which extracts an image-space generative field from the loss-gradient geometry of a pretrained discriminative model. DSF provides a training-free update rule that moves images toward model-consistent visual regions without retraining or task-specific targets. We apply DSF to diverse pretrained discriminative models, including ResNet classifiers, DETR detectors, and DINOv2 ViT encoders. DSF generates unconditional images from noise and supports conditional guidance, editing, inpainting, and explanation through the same field. DSF-generated samples also provide effective zero-shot quantization calibration when real images or labels are unavailable, preserving feature geometry…
PDF arXiv
arXiv cs.CLOct 14
MakSafe: A Spatiotemporal Semantic Framework for Formal Specification and Compliance Checking of Road Vehicle Safety Rules
Maksym Labzhaniia, Julien Deantoni, Marie-Agn{è}s Peraldi-Frati, Fr{é}d{é}ric Mallet
Scenario-based validation is essential for automated road vehicles but cannot cover the full variability of openworld traffic. We present MakSafe, a formal spatiotemporal framework for specifying and checking road vehicle safety rules through semantic zones, action-based forecasting, and trace-tree reasoning, avoiding uniform time discretization. We formalize the road network, vehicle states, and occupancy zones, then build executable rules for front-vehicle identification and timeto-collision (TTC) verification. The evaluation combines a tracedependent case study with a scalability benchmark over traffic density (N ), horizon (H), and branch factor (B). MakSafe yields branch-specific verdicts, deriving safe and unsafe outcomes from the same initial scene, and re-evaluates a scenario after a control update. A single rule check stays well below a millisecond across the reported sweep, and longer horizons slow the TTC check.
PDF arXiv
arXiv cs.LGOct 14
What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization
Julia Barth, Armin Sommer, Kunvar Thaman, Jannik Schilling
Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but does not pursue the intended goal. However, we still have a weak understanding of what happens to the intended goal in misgeneralizing agents. Across three Procgen environments and five robot tasks, we find three distinct forms of policy-goal misalignment: (1) functional target substitution, where behavior follows an alternative goal, (2) a policy shortcut, where behavior follows a learned action strategy, and (3) cue competition between intended and proxy goal cues. We distinguish these regimes by tracing where goal information remains available internally, how strongly it influences the policy, and what role it plays in action selection, and find that the goal is lost at different depths. Where the goal reaches the decision, as in cue competition, steering redirects the policy to the goal's location, raising intended-goal reach from 16.0% to 95.3%. Where it reaches the policy but not the decision, only new downstream wiring recovers goal reach, from 31.6% to 84.3% in the environment with the action shortcut. Where it is inert, neither works. Our results show that goal…
PDF arXiv
arXiv cs.CVCodeOct 14▲ 17
A Diagnostic Study of World-Space Human Recovery Under High Camera Dynamics
Hao-Jen Chien, Wei-Lun Chao, Yu-Lun Liu
Monocular world-space human recovery methods excel on current benchmarks, but these rarely include fast camera motion. We introduce the K-Pop stress test, broadcast dance videos that combine fast, large camera motion with dense multi-person choreography. Leading methods produce plausible per-frame meshes on these videos, yet their world-space trajectories drift far from the performers' stage positions. Through a systematic diagnosis, we show that the camera and human estimates each remain self-consistent, and that the drift appears when they are combined at mismatched scales. Each is recovered at its own scale, so a mismatch between the two can turn camera motion into human drift that grows with camera displacement and becomes severe in broadcast dance videos. We derive a closed-form correction that estimates this mismatch and applies one scale to all performers, removing the drift while keeping their relative positions in one shared world frame. It requires no retraining and works with existing backends. Combined with physical constraints, the correction yields stable world-space reconstruction for choreography modeling, scene editing, data generation, and virtual reshooting.…
PDF arXiv Code

Activity · 12 weeks

Fields · AI · updated 46m ago

Multimodal

Models that combine modalities such as vision-language models, video-language and any-to-any models.

584papers this week−20.0% vs last week

All papers · 3.4k With code · 225

Newest Most relevant

3.1k papers this month

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Slow and inefficient spatial reasoning in multimodal models2 papers raise it · 2 address it
Autoregressive decoding and answer parsing are costly for compact output spaces like spatial relations or coordinates.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Poor long-horizon consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across nonconsecutive video shots is difficult without shared references.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Limited fine-grained control and interaction in 3D spatial reasoning1 paper raises it · 0 address it
Models struggle with precise 3D-to-2D projection alignment and sensitive interaction history in spatial tasks.
- RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignmentraises
Scalable and realistic training environments for embodied agents1 paper raises it · 1 addresses it
Building faithful and realistic interactive worlds for agent training remains a bottleneck.
- AgentGarten: Code Worlds for Evolving Agentsraises · addresses

Named by an AI from these papers' abstracts.