Papers

Field

Top tools here

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Slow and inefficient spatial reasoning in multimodal models2 papers raise it · 2 address it
Autoregressive decoding and answer parsing are costly for compact output spaces like spatial relations or coordinates.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Poor long-horizon consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across nonconsecutive video shots is difficult without shared references.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Limited fine-grained control and interaction in 3D spatial reasoning1 paper raises it · 0 address it
Models struggle with precise 3D-to-2D projection alignment and sensitive interaction history in spatial tasks.
- RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignmentraises
Scalable and realistic training environments for embodied agents1 paper raises it · 1 addresses it
Building faithful and realistic interactive worlds for agent training remains a bottleneck.
- AgentGarten: Code Worlds for Evolving Agentsraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

video understanding · video language · video generation309 papers56 this week
autoencoders · imagenet · autoencoder233 papers47 this week
visual question answering · vision language models · vision language170 papers47 this week
video generation · efficient video · autoregressive video189 papers37 this week
semantic segmentation · vision language · attention208 papers34 this week
visual reasoning · multimodal reasoning · visual question answering179 papers32 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Activity

Recent papers

All papers in Research

arXiv cs.CVOct 14
SonoWorld: Towards Interactive Pose-Controllable Ultrasound World Models
Zefan Yang, Zheyuan Zhang, Boris Maihe, Yikang Liu, Hang Yu +5
Robotic ultrasound can standardize image acquisition and reduce operator workload and variability, but policy learning remains limited by the lack of realistic, controllable simulation environments. While controllable world models have advanced rapidly in natural-image domains, ultrasound world modeling remains challenging because each frame captures only a cross-section of unobserved 3D anatomy, causing small probe motions to induce large appearance changes. Existing approaches rely on robot-tracked poses and latent-space prediction, but do not directly generate the observations required for interactive control. We propose SonoWorld, a causal ultrasound world model that learns video dynamics from unlabeled clinical videos and grounds them to 6-DoF control using a small set of freehand clinical scans tracked with an EM sensor. SonoWorld is trained in three stages: pretraining a latent-action-conditioned video diffusion model on around 6.9 million unlabeled multi-organ ultrasound frames; grounding the learned video dynamics to metric probe poses with a small EM-tracked thyroid dataset; and applying distribution matching distillation to enable causal, long-horizon autoregressive…
PDF arXiv
arXiv cs.CVOct 13
Data-free Multistep Diffusion Distillation
Ngoc Duy Tran, Kien Do, Duc Thanh Nguyen, Truyen Tran
One-step diffusion distillation achieves strong performance on small datasets but remains insufficient for high-resolution or complex text-to-image settings. Extending to multistep distillation often requires training data, suffers from early-step degradation, inconsistent outputs across sampling budgets, and incurs high computational cost. We identify that these limitations stem from simultaneous training, where a single adapted network approximates multiple score functions of different student distributions at once, causing biased distribution matching and unstable optimization. Therefore, we propose Progressive Multistep Data-free Distillation (PMDD), which assigns a dedicated adapted network per student distribution while avoiding the memory overhead of maintaining multiple adapted networks. Combined with a stage-wise unforget loss to preserve earlier-step generations, PMDD reduces optimization bias in the student, provides faster convergence, and a single unified model that supports multiple inference budgets. Extensive experiments show that PMDD matches or outperforms data-free few-step baselines at every multistep budget, reaches FID 1.99 on ImageNet 64x64 in two steps,…
PDF arXiv
arXiv cs.ROOct 13
MIND: Instruction-Conditioned Incremental Grounding for Robots
Mohamed Abouagour, Junhyun Nam, Wonse Jo, Byung-Cheol Min
Robots following persistent language instructions must decide which cached room judgments remain valid as their environment changes. We present MIND (Maintenance via INstruction-conditioned Dependencies), a framework for maintaining target-room decisions without re-evaluating every room after each event. A language model compiles each instruction into dependencies for semantic qualification and target selection. The controller intersects semantic dependencies with event footprints, refreshes affected judgments, and updates reachability and selection separately. For fixed deterministic evaluation, planning, and selection, conservative dependency and footprint coverage preserves the decisions of full recomputation. On 240 five-event ProcTHOR sequences with an exact benchmark predicate, MIND reduces mean semantic evaluations from 30.6 to 11.2 while preserving all 1,200 post-event decisions. Frozen-evaluator experiments with Phi-4 and Qwen3-14B also preserve all 1,200 decisions, without improving the evaluators' underlying task accuracy. In 32 closed-loop AI2-THOR episodes using exact-predicate qualification and metadata-derived footprints, MIND matches the final target decision of…
PDF arXiv
arXiv cs.CVOct 13
Long-horizon Video Generation with Slot Memory Addressed By Region Tokens
Eren Cetin, Julius Erbach, Vukasin Bozic, Hyojun Go, Zixiang Zhao +4
Memory for long-horizon video generation must satisfy two competing requirements: it must remain bounded so generation can continue indefinitely, while still recovering what was observed at a place when the camera returns, regardless of how long ago it was seen. Reconstruction-based memories provide spatial recall by caching and reprojecting explicit geometry, but require growing storage and depend on estimated 3D structure. Recurrent memories keep a constant-size state, but retention is governed by temporal recurrence, causing old observations to fade with time. We introduce SMART, a slot memory addressed by region tokens, which combines bounded storage with spatially addressed recall. SMART stores content in a constant-size slot recurrence whose writes are addressed by the observing camera and whose reads are queried by the rendering camera, without constructing an explicit 3D representation or maintaining a growing cache. Consequently, recall is conditioned on inter-view geometry rather than on recency. Each read additionally provides a confidence estimate that conditions a chunk-causal diffusion transformer, allowing it to selectively trust retrieved content while relying on…
PDF arXiv
arXiv cs.AIOct 13
Write Once, Plan Many: Building Reusable Visual Planning Systems from One Scene
Ruixiao Yang, Yilun Hao, Mingxin Yu, Yang Zhang, Chuchu Fan
Vision-language models (VLMs) can reason about visual planning tasks, but querying them for every instance incurs repeated inference cost and latency. Generating reusable planning programs offers an alternative, but constructing reliable planning systems for new problem domains without extensive supervision remains challenging. We present a training-free framework that constructs reusable perception and planning programs from a domain specification and a single practice scene without a paired solution. The central challenge is obtaining useful feedback from the limited input. Our framework addresses it by leveraging VLM's ability to provide a renderer and a dynamic model based on its scene understanding. Rendering detected states enables comparison with the practice scene, while checking simulated action outcomes against the domain specification helps expose errors in rule implementation. Together, these checks guide refinement of the grounding program and planner without requiring solution labels. After refinement and component selection, the programs are frozen and solve new instances without further VLM calls. The same framework constructs systems across domains without manual…
PDF arXiv
arXiv cs.CVOct 13
A Gradient-Regularized Deep Learning Method for Restoring Image in Myopic Vision
Kebin Contreras, Luis Toscano-Palomino, Rafael Torres, Jorge Bacca
Myopic vision degrades the quality of human vision due to spherical eye defocus aberrations that impair the perception of fine spatial details. To address this limitation, this work proposes a deep learning-based approach to computationally design transformed images that allow individuals with myopia in fixed distance to perceive scenes with enhanced clarity. The method trains a neural network to counteract defocus aberrations modeled using the second-order Zernike polynomial $Z_2^0$. The approach employs a U-Net architecture trained with a loss function combining reconstruction error minimization with spatial difference structural preservation through $\ell_1$ regularization. Training is carried out exclusively on the KITTI dataset, while testing is performed on DIV2K and ImageNet datasets, demonstrating generalization across diverse image distributions. Under myopic conditions modeled by $Z_2^0$, baseline performance shows visual degradation with PSNR values ranging from 19.47 to 27.42 dB across datasets and severity levels. The proposed gradient-regularized method achieves consistent improvements, delivering PSNR gains ranging from 1.48 to 7.34 dB and significantly lower…
PDF arXiv

Papers

Topics in this field

video understanding · video language · video generation309 papers56 this week
autoencoders · imagenet · autoencoder233 papers47 this week
visual question answering · vision language models · vision language170 papers47 this week
video generation · efficient video · autoregressive video189 papers37 this week
semantic segmentation · vision language · attention208 papers34 this week
visual reasoning · multimodal reasoning · visual question answering179 papers32 this week

Active contributors · 90 days

Recent papers

All papers in Research

arXiv cs.CVOct 14
SonoWorld: Towards Interactive Pose-Controllable Ultrasound World Models
Zefan Yang, Zheyuan Zhang, Boris Maihe, Yikang Liu, Hang Yu +5
Robotic ultrasound can standardize image acquisition and reduce operator workload and variability, but policy learning remains limited by the lack of realistic, controllable simulation environments. While controllable world models have advanced rapidly in natural-image domains, ultrasound world modeling remains challenging because each frame captures only a cross-section of unobserved 3D anatomy, causing small probe motions to induce large appearance changes. Existing approaches rely on robot-tracked poses and latent-space prediction, but do not directly generate the observations required for interactive control. We propose SonoWorld, a causal ultrasound world model that learns video dynamics from unlabeled clinical videos and grounds them to 6-DoF control using a small set of freehand clinical scans tracked with an EM sensor. SonoWorld is trained in three stages: pretraining a latent-action-conditioned video diffusion model on around 6.9 million unlabeled multi-organ ultrasound frames; grounding the learned video dynamics to metric probe poses with a small EM-tracked thyroid dataset; and applying distribution matching distillation to enable causal, long-horizon autoregressive…
PDF arXiv
arXiv cs.CVOct 13
Data-free Multistep Diffusion Distillation
Ngoc Duy Tran, Kien Do, Duc Thanh Nguyen, Truyen Tran
One-step diffusion distillation achieves strong performance on small datasets but remains insufficient for high-resolution or complex text-to-image settings. Extending to multistep distillation often requires training data, suffers from early-step degradation, inconsistent outputs across sampling budgets, and incurs high computational cost. We identify that these limitations stem from simultaneous training, where a single adapted network approximates multiple score functions of different student distributions at once, causing biased distribution matching and unstable optimization. Therefore, we propose Progressive Multistep Data-free Distillation (PMDD), which assigns a dedicated adapted network per student distribution while avoiding the memory overhead of maintaining multiple adapted networks. Combined with a stage-wise unforget loss to preserve earlier-step generations, PMDD reduces optimization bias in the student, provides faster convergence, and a single unified model that supports multiple inference budgets. Extensive experiments show that PMDD matches or outperforms data-free few-step baselines at every multistep budget, reaches FID 1.99 on ImageNet 64x64 in two steps,…
PDF arXiv
arXiv cs.ROOct 13
MIND: Instruction-Conditioned Incremental Grounding for Robots
Mohamed Abouagour, Junhyun Nam, Wonse Jo, Byung-Cheol Min
Robots following persistent language instructions must decide which cached room judgments remain valid as their environment changes. We present MIND (Maintenance via INstruction-conditioned Dependencies), a framework for maintaining target-room decisions without re-evaluating every room after each event. A language model compiles each instruction into dependencies for semantic qualification and target selection. The controller intersects semantic dependencies with event footprints, refreshes affected judgments, and updates reachability and selection separately. For fixed deterministic evaluation, planning, and selection, conservative dependency and footprint coverage preserves the decisions of full recomputation. On 240 five-event ProcTHOR sequences with an exact benchmark predicate, MIND reduces mean semantic evaluations from 30.6 to 11.2 while preserving all 1,200 post-event decisions. Frozen-evaluator experiments with Phi-4 and Qwen3-14B also preserve all 1,200 decisions, without improving the evaluators' underlying task accuracy. In 32 closed-loop AI2-THOR episodes using exact-predicate qualification and metadata-derived footprints, MIND matches the final target decision of…
PDF arXiv
arXiv cs.CVOct 13
Long-horizon Video Generation with Slot Memory Addressed By Region Tokens
Eren Cetin, Julius Erbach, Vukasin Bozic, Hyojun Go, Zixiang Zhao +4
Memory for long-horizon video generation must satisfy two competing requirements: it must remain bounded so generation can continue indefinitely, while still recovering what was observed at a place when the camera returns, regardless of how long ago it was seen. Reconstruction-based memories provide spatial recall by caching and reprojecting explicit geometry, but require growing storage and depend on estimated 3D structure. Recurrent memories keep a constant-size state, but retention is governed by temporal recurrence, causing old observations to fade with time. We introduce SMART, a slot memory addressed by region tokens, which combines bounded storage with spatially addressed recall. SMART stores content in a constant-size slot recurrence whose writes are addressed by the observing camera and whose reads are queried by the rendering camera, without constructing an explicit 3D representation or maintaining a growing cache. Consequently, recall is conditioned on inter-view geometry rather than on recency. Each read additionally provides a confidence estimate that conditions a chunk-causal diffusion transformer, allowing it to selectively trust retrieved content while relying on…
PDF arXiv
arXiv cs.AIOct 13
Write Once, Plan Many: Building Reusable Visual Planning Systems from One Scene
Ruixiao Yang, Yilun Hao, Mingxin Yu, Yang Zhang, Chuchu Fan
Vision-language models (VLMs) can reason about visual planning tasks, but querying them for every instance incurs repeated inference cost and latency. Generating reusable planning programs offers an alternative, but constructing reliable planning systems for new problem domains without extensive supervision remains challenging. We present a training-free framework that constructs reusable perception and planning programs from a domain specification and a single practice scene without a paired solution. The central challenge is obtaining useful feedback from the limited input. Our framework addresses it by leveraging VLM's ability to provide a renderer and a dynamic model based on its scene understanding. Rendering detected states enables comparison with the practice scene, while checking simulated action outcomes against the domain specification helps expose errors in rule implementation. Together, these checks guide refinement of the grounding program and planner without requiring solution labels. After refinement and component selection, the programs are frozen and solve new instances without further VLM calls. The same framework constructs systems across domains without manual…
PDF arXiv
arXiv cs.CVOct 13
A Gradient-Regularized Deep Learning Method for Restoring Image in Myopic Vision
Kebin Contreras, Luis Toscano-Palomino, Rafael Torres, Jorge Bacca
Myopic vision degrades the quality of human vision due to spherical eye defocus aberrations that impair the perception of fine spatial details. To address this limitation, this work proposes a deep learning-based approach to computationally design transformed images that allow individuals with myopia in fixed distance to perceive scenes with enhanced clarity. The method trains a neural network to counteract defocus aberrations modeled using the second-order Zernike polynomial $Z_2^0$. The approach employs a U-Net architecture trained with a loss function combining reconstruction error minimization with spatial difference structural preservation through $\ell_1$ regularization. Training is carried out exclusively on the KITTI dataset, while testing is performed on DIV2K and ImageNet datasets, demonstrating generalization across diverse image distributions. Under myopic conditions modeled by $Z_2^0$, baseline performance shows visual degradation with PSNR values ranging from 19.47 to 27.42 dB across datasets and severity levels. The proposed gradient-regularized method achieves consistent improvements, delivering PSNR gains ranging from 1.48 to 7.34 dB and significantly lower…
PDF arXiv

Activity · 12 weeks

Fields · AI · updated 2h ago

Multimodal

Models that combine modalities such as vision-language models, video-language and any-to-any models.

580papers this week−19.9% vs last week

All papers · 3.4k With code · 227

Newest Most relevant

3.1k papers this month

Open problems in Multimodal

Lack of holistic video understanding benchmarks1 paper raises it · 1 addresses it
Existing benchmarks cover only partial evaluation dimensions, limiting systematic diagnosis of omni-modal video understanding.
- OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understandingraises · addresses
Slow and inefficient spatial reasoning in multimodal models2 papers raise it · 2 address it
Autoregressive decoding and answer parsing are costly for compact output spaces like spatial relations or coordinates.
- VideoJev: System One Model for Video Understanding and Interactionraises · addresses
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matchingraises · addresses
Poor long-horizon consistency in agentic video generation1 paper raises it · 1 addresses it
Maintaining spatial and temporal consistency across nonconsecutive video shots is difficult without shared references.
- Any Time, Any Space: A World-State-Centric Framework for Agentic Video Generationraises · addresses
Limited fine-grained control and interaction in 3D spatial reasoning1 paper raises it · 0 address it
Models struggle with precise 3D-to-2D projection alignment and sensitive interaction history in spatial tasks.
- RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignmentraises
Scalable and realistic training environments for embodied agents1 paper raises it · 1 addresses it
Building faithful and realistic interactive worlds for agent training remains a bottleneck.
- AgentGarten: Code Worlds for Evolving Agentsraises · addresses

Named by an AI from these papers' abstracts.