Papers

Field

Top tools here

Open problems in Image & video generation

3D consistency in generated videos1 paper raises it · 2 address it
Generated videos often lack coherent 3D structure, limiting their use in interactive or simulation applications.
- Orbia: Do Generated Videos Form a Consistent 3D World?raises · addresses
- InfiniSplat v2: Analyzing and Improving Feed-Forward 3DGS from Any Viewsaddresses
Stable multi-turn image editing1 paper raises it · 1 addresses it
Repeated edits degrade image quality due to accumulating distribution shifts between training and inference.
- ACEdit: Stable Multi-Turn Image Editing via Asymmetric Conditioningraises · addresses
Efficient autoregressive video generation2 papers raise it · 3 address it
Uniform denoising for all tokens wastes computation, slowing down streaming or interactive generation.
- Speculative Forcing for Efficient Autoregressive Video Generationraises · addresses
- Glance Forcing: Making Bidirectional Diffusion Autoregressive at a Glanceraises · addresses
- Speculative Sampling in Interactive Video Generationaddresses
Semantic alignment in text-to-image prompt engineering1 paper raises it · 1 addresses it
Token-level rewards ignore semantic spans, reducing the effectiveness of RL-based prompt optimization.
- SCOPE: Semantic Bounded Credit Optimization for Prompt Engineering in Text-to-image Generationraises · addresses
Unified multimodal representation for generation1 paper raises it · 1 addresses it
Discrete, continuous, and hybrid visual tokenizers struggle to balance representational capacity with alignment to text modeling.
- Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generationraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.GROct 15
From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches
Ahmet Neçirvan Doğan, Eren Acar Başaran, Metin Sezgin, Zahra Montazeri
Sketching is one of the most natural ways for people to express visual ideas. Unlike the text prompts that most generative systems rely on today, a sketch captures spatial structure and intent directly, including where a designer wants highlights and shadows to appear. Yet current sketch-to-3D pipelines have no input for that intent: they bake the drawn shading into the texture as dark paint, and the light cannot be moved. We propose a system that transforms a rough garment sketch, marked with sparse annotations (a star for a highlight and a moon for a shadow), and a short text prompt describing the texture, into a textured 3D mesh with an explicit directional light, as a scene that can be relit and edited after generation. The two symbols are the most frequent proposals in an elicitation study with art students. From the marked sketch the system predicts the scene's directional light and, separately, a displacement field that bends the surface so that the moon-marked regions fall into shadow under that light, rather than being painted dark in the texture.
PDF arXiv
arXiv cs.CVOct 15
Seeing Both the Forest and the Trees: Reusing Holistic 3D Priors for Part-Decomposed Generation
Yiqun Zhao, Binbin Huang, Haobin Duan, Zibo Zhao, Shenghua Gao
Generating 3D assets with explicit parts requires two properties at once: geometric detail inside each part and structural coherence across the whole object. Recent image-conditioned 3D generation models adopt a structured 3D representation that anchors tokens to explicit spatial positions, capturing both properties, but their output remains a single fused mesh. We ask how a part-decomposed generation model can inherit both properties of these holistic priors, and answer with STRUCTPARTS, a framework whose core technical contribution is a dual-frame coordinate interleaving mechanism: each shape latent is interpreted under both a part-local and an object-global reference frame, and processed through a global-local inter-leaved transformer that simultaneously reuses the prior's geometric detail at the part scale and its structural coherence at the whole-object scale. As its structural input, STRUCTPARTS uses a segmented mesh scaffold, a coarse mesh paired with a face-level segmentation mask kept entirely within the prior's native 3D representation. With only lightweight finetuning, both intra-part detail and inter-part coherence emerge from the same pretrained weights. On…
PDF arXiv
arXiv cs.CVOct 15
Edit4D: Editable 4D Scene Generation with Textual Control
Chanwoo Kim, Tackgeun You, Hwasup Lim
We introduce Edit4D, an editable text-to-4D scene generation framework that constructs 4D scenes from separate background and dynamic character representations. Recent advances in generative models have enabled text-driven dynamic scene synthesis, but editable 4D scene generation remains challenging due to the need to preserve spatial geometry and temporal dynamics. Existing approaches typically generate holistic scene videos, where the background and dynamic characters are coupled, making object-level editing and flexible camera control difficult. Edit4D addresses these limitations through a stage-specific generation pipeline. Given a user instruction, Edit4D first decomposes it into stage-specific prompts for background, dynamic character, and scene-character composition. Using these prompts, we synthesize multi-view scene images with text-to-image and camera-conditioned generative priors and reconstruct a background 3D Gaussian splatting representation. We then generate a character image, animate it with an image-conditioned video prior, reconstruct a 4D mesh, and bind Gaussian primitives to obtain a character representation. Finally, both components are composed through…
PDF arXiv
arXiv cs.CVOct 15
Anchor3DGS: Feed-Forward 3D Gaussian Splatting with Compact Anchor-Based Representation
Sheng Ye, Jiangke Lin, Fudong Wang, Yi Yuan, Zhen-Hui Dong +1
Recent feed-forward 3D Gaussian Splatting (3DGS) methods have enabled effective multi-view 3D reconstruction by predicting Gaussian primitives in a single forward pass. However, existing approaches predominantly adopt a pixel-aligned paradigm that predicts one Gaussian per input pixel, coupling the Gaussian count to the input resolution and number of views. This also leads to redundant primitives in textureless regions and insufficient coverage in geometrically complex areas. We present Anchor3DGS, a pose-free feed-forward 3DGS framework that replaces pixel-aligned prediction with a compact anchor-based representation. Guided by information entropy, our method places a set of 3D anchors in the scene, concentrating them in regions of higher visual complexity while maintaining broad spatial coverage. Each anchor performs occlusion-aware aggregation of multi-view image features, exchanges information with other anchors, and is decoded into a set of local Gaussians. A lightweight feature enhancement module further refines the coarse RGB output from Gaussian splatting and improves rendering fidelity. Extensive experiments on diverse benchmark datasets demonstrate that our Anchor3DGS…
PDF arXiv
arXiv cs.ROOct 15
Training-Free Target-Aware Refinement of 3D Gaussian Language Map
Sibaek Lee, Kyeongsu Kang, Seongbo Ha, Yewon Jeon, Joonyeol Choi +7
Distilling vision-language features into 3D Gaussian Splatting yields a language map that can be queried with arbitrary text. In deployment, however, a small set of user-specified targets must be reliably located, which current distilled maps often miss, while open-vocabulary queryability for other words must remain intact. We propose a training-free, target-aware refinement of an already-distilled language map. Guided by a text-promptable segmenter, the visual embedding inside each detected target region is blended with the target text embedding, a convex combination whose coefficient sets the amount of injected prior, while non-target regions are left untouched. Injecting too much overwrites appearance cues and general queryability, so we seek, for each target, the smallest coefficient that still lets it win, and we obtain every coefficient in closed form, leaving no hyperparameters to tune. We further identify a failure mode of injection in language-goal localization. Text embeddings share a common component, and injecting them makes the standard cosine read-out query-agnostic, which a centering step in the read-out removes. Across three benchmarks, the refinement consistently…
PDF arXiv
arXiv cs.CVOct 15
Decoupled Safety Control: A Safety-Control Algorithm for Training-Free Safety Guidance
Huilin Zhou, Ruoxi Cheng, Yuhang Wang, Yuming Liu, Minghao Sun +2
Diffusion models achieve strong performance in text-to-image generation, but they can still produce unsafe content. Training-free safety guidance offers a practical inference-time alternative that intervenes during sampling without modifying model weights. However, many existing safeguards still rely on a coupled safety-control chain. In this chain, the same unsafe condition simultaneously specifies what should be avoided, indicates the current unsafe signal, constructs the intervention, and determines when that intervention remains active. This coupling can work when unsafe factors are explicit, but becomes less reliable once harmful and benign semantics are entangled in the current generation state, often leaving residual unsafe content or unnecessarily degrading benign-generation quality or prompt fidelity. We address this problem with Decoupled Safety Control (DSC), a plug-in safety-control algorithm for training-free safety guidance. DSC reformulates training-free safety guidance as an explicit state-dependent control process, so the executed intervention is determined from the current generation state rather than inherited directly from a single unsafe condition.…
PDF arXiv

Papers

Recent papers

All in Research →

arXiv cs.GROct 15
From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches
Ahmet Neçirvan Doğan, Eren Acar Başaran, Metin Sezgin, Zahra Montazeri
Sketching is one of the most natural ways for people to express visual ideas. Unlike the text prompts that most generative systems rely on today, a sketch captures spatial structure and intent directly, including where a designer wants highlights and shadows to appear. Yet current sketch-to-3D pipelines have no input for that intent: they bake the drawn shading into the texture as dark paint, and the light cannot be moved. We propose a system that transforms a rough garment sketch, marked with sparse annotations (a star for a highlight and a moon for a shadow), and a short text prompt describing the texture, into a textured 3D mesh with an explicit directional light, as a scene that can be relit and edited after generation. The two symbols are the most frequent proposals in an elicitation study with art students. From the marked sketch the system predicts the scene's directional light and, separately, a displacement field that bends the surface so that the moon-marked regions fall into shadow under that light, rather than being painted dark in the texture.
PDF arXiv
arXiv cs.CVOct 15
Seeing Both the Forest and the Trees: Reusing Holistic 3D Priors for Part-Decomposed Generation
Yiqun Zhao, Binbin Huang, Haobin Duan, Zibo Zhao, Shenghua Gao
Generating 3D assets with explicit parts requires two properties at once: geometric detail inside each part and structural coherence across the whole object. Recent image-conditioned 3D generation models adopt a structured 3D representation that anchors tokens to explicit spatial positions, capturing both properties, but their output remains a single fused mesh. We ask how a part-decomposed generation model can inherit both properties of these holistic priors, and answer with STRUCTPARTS, a framework whose core technical contribution is a dual-frame coordinate interleaving mechanism: each shape latent is interpreted under both a part-local and an object-global reference frame, and processed through a global-local inter-leaved transformer that simultaneously reuses the prior's geometric detail at the part scale and its structural coherence at the whole-object scale. As its structural input, STRUCTPARTS uses a segmented mesh scaffold, a coarse mesh paired with a face-level segmentation mask kept entirely within the prior's native 3D representation. With only lightweight finetuning, both intra-part detail and inter-part coherence emerge from the same pretrained weights. On…
PDF arXiv
arXiv cs.CVOct 15
Edit4D: Editable 4D Scene Generation with Textual Control
Chanwoo Kim, Tackgeun You, Hwasup Lim
We introduce Edit4D, an editable text-to-4D scene generation framework that constructs 4D scenes from separate background and dynamic character representations. Recent advances in generative models have enabled text-driven dynamic scene synthesis, but editable 4D scene generation remains challenging due to the need to preserve spatial geometry and temporal dynamics. Existing approaches typically generate holistic scene videos, where the background and dynamic characters are coupled, making object-level editing and flexible camera control difficult. Edit4D addresses these limitations through a stage-specific generation pipeline. Given a user instruction, Edit4D first decomposes it into stage-specific prompts for background, dynamic character, and scene-character composition. Using these prompts, we synthesize multi-view scene images with text-to-image and camera-conditioned generative priors and reconstruct a background 3D Gaussian splatting representation. We then generate a character image, animate it with an image-conditioned video prior, reconstruct a 4D mesh, and bind Gaussian primitives to obtain a character representation. Finally, both components are composed through…
PDF arXiv
arXiv cs.CVOct 15
Anchor3DGS: Feed-Forward 3D Gaussian Splatting with Compact Anchor-Based Representation
Sheng Ye, Jiangke Lin, Fudong Wang, Yi Yuan, Zhen-Hui Dong +1
Recent feed-forward 3D Gaussian Splatting (3DGS) methods have enabled effective multi-view 3D reconstruction by predicting Gaussian primitives in a single forward pass. However, existing approaches predominantly adopt a pixel-aligned paradigm that predicts one Gaussian per input pixel, coupling the Gaussian count to the input resolution and number of views. This also leads to redundant primitives in textureless regions and insufficient coverage in geometrically complex areas. We present Anchor3DGS, a pose-free feed-forward 3DGS framework that replaces pixel-aligned prediction with a compact anchor-based representation. Guided by information entropy, our method places a set of 3D anchors in the scene, concentrating them in regions of higher visual complexity while maintaining broad spatial coverage. Each anchor performs occlusion-aware aggregation of multi-view image features, exchanges information with other anchors, and is decoded into a set of local Gaussians. A lightweight feature enhancement module further refines the coarse RGB output from Gaussian splatting and improves rendering fidelity. Extensive experiments on diverse benchmark datasets demonstrate that our Anchor3DGS…
PDF arXiv
arXiv cs.ROOct 15
Training-Free Target-Aware Refinement of 3D Gaussian Language Map
Sibaek Lee, Kyeongsu Kang, Seongbo Ha, Yewon Jeon, Joonyeol Choi +7
Distilling vision-language features into 3D Gaussian Splatting yields a language map that can be queried with arbitrary text. In deployment, however, a small set of user-specified targets must be reliably located, which current distilled maps often miss, while open-vocabulary queryability for other words must remain intact. We propose a training-free, target-aware refinement of an already-distilled language map. Guided by a text-promptable segmenter, the visual embedding inside each detected target region is blended with the target text embedding, a convex combination whose coefficient sets the amount of injected prior, while non-target regions are left untouched. Injecting too much overwrites appearance cues and general queryability, so we seek, for each target, the smallest coefficient that still lets it win, and we obtain every coefficient in closed form, leaving no hyperparameters to tune. We further identify a failure mode of injection in language-goal localization. Text embeddings share a common component, and injecting them makes the standard cosine read-out query-agnostic, which a centering step in the read-out removes. Across three benchmarks, the refinement consistently…
PDF arXiv
arXiv cs.CVOct 15
Decoupled Safety Control: A Safety-Control Algorithm for Training-Free Safety Guidance
Huilin Zhou, Ruoxi Cheng, Yuhang Wang, Yuming Liu, Minghao Sun +2
Diffusion models achieve strong performance in text-to-image generation, but they can still produce unsafe content. Training-free safety guidance offers a practical inference-time alternative that intervenes during sampling without modifying model weights. However, many existing safeguards still rely on a coupled safety-control chain. In this chain, the same unsafe condition simultaneously specifies what should be avoided, indicates the current unsafe signal, constructs the intervention, and determines when that intervention remains active. This coupling can work when unsafe factors are explicit, but becomes less reliable once harmful and benign semantics are entangled in the current generation state, often leaving residual unsafe content or unnecessarily degrading benign-generation quality or prompt fidelity. We address this problem with Decoupled Safety Control (DSC), a plug-in safety-control algorithm for training-free safety guidance. DSC reformulates training-free safety guidance as an explicit state-dependent control process, so the executed intervention is determined from the current generation state rather than inherited directly from a single unsafe condition.…
PDF arXiv

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 51m ago

Image & video generation

Generating or editing images, video or 3D with diffusion, autoregressive or other generative models.

39papers this week

All papers · 39 With code · 8

Newest Most relevant

39 papers this month

Active contributors · 90 days

Open problems in Image & video generation

3D consistency in generated videos1 paper raises it · 2 address it
Generated videos often lack coherent 3D structure, limiting their use in interactive or simulation applications.
- Orbia: Do Generated Videos Form a Consistent 3D World?raises · addresses
- InfiniSplat v2: Analyzing and Improving Feed-Forward 3DGS from Any Viewsaddresses
Stable multi-turn image editing1 paper raises it · 1 addresses it
Repeated edits degrade image quality due to accumulating distribution shifts between training and inference.
- ACEdit: Stable Multi-Turn Image Editing via Asymmetric Conditioningraises · addresses
Efficient autoregressive video generation2 papers raise it · 3 address it
Uniform denoising for all tokens wastes computation, slowing down streaming or interactive generation.
- Speculative Forcing for Efficient Autoregressive Video Generationraises · addresses
- Glance Forcing: Making Bidirectional Diffusion Autoregressive at a Glanceraises · addresses
- Speculative Sampling in Interactive Video Generationaddresses
Semantic alignment in text-to-image prompt engineering1 paper raises it · 1 addresses it
Token-level rewards ignore semantic spans, reducing the effectiveness of RL-based prompt optimization.
- SCOPE: Semantic Bounded Credit Optimization for Prompt Engineering in Text-to-image Generationraises · addresses
Unified multimodal representation for generation1 paper raises it · 1 addresses it
Discrete, continuous, and hybrid visual tokenizers struggle to balance representational capacity with alignment to text modeling.
- Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generationraises · addresses

Named by an AI from these papers' abstracts.