Papers

Field

Top tools here

Open problems in Computer vision

View-dependent geometry features in 3D world models1 paper raises it · 1 addresses it
Entangles scene dynamics with camera motion, making state prediction harder and less generalizable.
- GeoState: Learning View-Decoupled Geometry States for 3D World Modelingraises · addresses
High-quality 3D Gaussian rendering under large viewpoint changes1 paper raises it · 1 addresses it
Feed-forward methods struggle to learn Gaussian attributes for consistent rendering across extreme viewpoints.
- InfiniSplat v2: Analyzing and Improving Feed-Forward 3DGS from Any Viewsraises · addresses
Dense 4D human correspondence without templates1 paper raises it · 1 addresses it
Existing methods fail to track surface points across articulation, deformation, and occlusion without relying on body templates.
- Dense4D: Pixel-Level 4D Human Correspondenceraises · addresses
Open-set human activity and attribute description in videos1 paper raises it · 1 addresses it
Current datasets and models simplify to closed-set or single-person tasks, limiting real-world applicability.
- Grounded Human-Attributed Description and Activity Recognition in Videosraises · addresses
Annotation-free medical image segmentation1 paper raises it · 1 addresses it
Manual annotations for tasks like retinal vessel segmentation are time-consuming and require expert knowledge.
- FPG-Seg: Annotation-free Retinal Vessel Topology Segmentationraises · addresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.CVOct 15
FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities
Xin Zhang, Robby T. Tan
Multi-modal semantic segmentation benefits from complementary sensors, but this benefit is fragile when some modalities are degraded or missing at test time. Many existing methods rely on dedicated fusion modules or multi-branch cross-modal encoders, and the fusion often becomes unreliable when modalities are absent, leading to substantial performance drops. We propose \textbf{FuseAdapt}, a parameter-efficient framework that performs fusion within the adaptation space of a frozen vision foundation model (VFM), exploring whether VFM priors can serve as an effective shared backbone for multi-modal segmentation. Instead of attaching a standalone fusion network, FuseAdapt assigns each modality a lightweight adaptation branch and performs \emph{adaptation-space fusion} by keeping Q/V low-rank adaptation private to each modality and routing only FFN low-rank residual activations across modalities inside the frozen backbone. The pretrained VFM remains fixed while modality-specific corrections and explicit cross-modal interaction are learned through the adaptation paths. To handle missing modalities, we further introduce \textbf{Predictive Modality Modeling (PMM)}, a direct…
PDF arXiv
arXiv cs.CVOct 15
Mixture of Multi-Sequence Feature Difference for Contrast-Enhanced MRI Synthesis
Jonghun Kim, Junmo Kwon, Hyunjin Park
Contrast-enhanced magnetic resonance imaging (CE-MRI) is crucial for accurately diagnosing and segmenting brain tumors. However, the acquisition of CE-MRI requires injecting contrast agents, which poses potential health risks and increases scan costs. Here, we propose a CE-MRI synthesis method from non-contrast sequences, reducing both risk and cost. Unlike existing methods that rely on channel-wise concatenation to fuse multi-sequence information for synthesis, we introduce a Mixture of Feature Difference (MoFD) module. The proposed module uses a dynamic weighting strategy to adaptively select and aggregate informative features from both individual source sequences and explicit differences among them. Quantitative and qualitative evaluations demonstrate that our proposed method outperforms baselines in terms of image fidelity and perceptual quality on the UCSF-PDGM and UPenn-GBM datasets. Our method successfully reconstructs tumor regions and achieves segmentation performance comparable to that obtained with real CE-MRI. Our ablation studies show that although adversarial learning improves perceptual quality, a regression-based variant of our method achieves higher structural…
PDF arXiv
arXiv cs.CVOct 15
Bridging the Markerless Tracking Validation Gap: Intraoperative X-ray Workflow Integration
Connor Daly, Salvatore Russo, Jinendra Ekanayake, Daniel S. Elson, Ferdinando Rodriguez y Baena
Markerless intraoperative tracking of spinal anatomy via RGB-D sensors is a promising alternative to invasive bone-anchored fiducials, yet no existing clinical dataset provides independent subsurface ground truth for validating these methods. We address this gap with a clinically viable pipeline that derives per-vertebra 6-DoF pose from routine intraoperative C-arm fluoroscopy, co-registered to the RGB-D camera frame through a multimodal fiducial array detectable in both imaging modalities. Critically, the pipeline requires no additional radiation, instrumentation, or deviation from standard surgical workflows, relying solely on the fluoroscopic images already acquired during routine level checks. A calibrated transform chain links the vertebral pose recovered via multi-view 2D-3D CT-to-fluoroscopy registration to the RGB-D coordinate frame, enabling direct comparison of any markerless registration estimate against a subsurface reference. Ex vivo validation against optically tracked ovine vertebrae confirms end-to-end accuracy of 2.06 +- 0.29 mm and 3.35 +- 0.93 degrees, sufficient to serve as independent ground truth for methods with substantially larger errors. We demonstrate…
PDF arXiv
arXiv cs.LGOct 15
ShiftQAT: Quantization-Aware Training for the Numerical Contract of a Fixed-Function Edge CNN Accelerator
Oguzhan Buyuksolak, Osman Erman Okman, Mehmet Gorkem Ulkar
Fixed-function CNN accelerators make edge inference fast and energy-efficient, but they execute only networks that obey a precise numerical contract. On the MAX7800x family, this contract requires power-of-two, per-tensor, symmetric scales, 8-bit activations and biases, integer-only arithmetic, and one statically configured shift per layer. We present ShiftQAT, the quantization-aware training (QAT) flow we developed for this family, which produces networks satisfying the contract by construction. It is the default training path of ai8x-training, the family's open-source training toolchain. ShiftQAT resolves residual, concatenation, and weight-sharing dependencies so that activation scales propagate statically and each layer's requantization reduces to one offline-configured shift. It also trains weights and biases on a joint power-of-two grid that fits the 8-bit bias budget. Among the compared methods, only ShiftQAT satisfies all of the contract's scaling constraints, including signed 8-bit activation storage. Even so, on ImageNet it matches the W8A8 accuracy drop of less constrained QAT methods and stays within 2.9 percentage points of FP32 at W4A8. On MAX78002 silicon,…
PDF arXiv
arXiv cs.CVOct 15
SCMerge: Spatial-Consistency-Aware Token Pair Merging for Efficient Semantic Segmentation
Junfeng Ren, Yi Gong, Yuan Zeng
Vision Transformer architectures have showcased impressive performance on semantic segmentation, yet their quadratic attention complexity hinders practical applications. Recent works have focused on merging tokens to reduce sequence length and thus improve transformer efficiency. However, existing token merging methods primarily rely on feature similarity to identify similar tokens, overlooking spatial consistency, which may introduce segmentation errors in semantic segmentation. In this work, we propose SCMerge, a training-free token merging method that jointly considers feature similarity and spatial consistency in pairing similar tokens. The pairwise tokens are then merged within the encoder, and the resulting token sequences are reconstructed onto the original patch grid after decoding. The proposed method is flexible and can be plugged into existing vision Transformer-based semantic segmentation pipelines. Extensive experiments on the benchmark datasets ADE20K, Pascal Context, and Cityscapes show that SCMerge consistently achieves a better trade-off between segmentation accuracy and efficiency than baseline methods.
PDF arXiv
arXiv cs.CVOct 15
Spurious Attention Heads in Vision Transformers: Mechanistic Identification of Annotation-Driven Bias in Medical Chest X-Ray Analysis
Joshua Muhumuza, Cornelius Kalenzi
Vision Transformers deployed in medical imaging exploit spurious correlations, yet the internal mechanisms remain unidentified and impossible to fix. We apply mechanistic interpretability to locate one such mechanism in ViT-Base 16: a single attention head (L8H10) that functions as a dedicated annotation-reader on chest X-rays, attending to corner text rather than lung pathology in 62% of 200 COVID-19 images. Causal validation via inpainting confirms annotation content rather than luminance or edge density drives the behaviour, increasing lung attention 1.52 times upon removal in affected images ($p{<}0.0001$). The finding replicates across NIH ChestX-ray14 (79%), DeiT-Base (88%), and ViT-Large 16 (80%), establishing spurious annotation-attending heads as a consistent property of ViT-family architectures across two datasets and three architectures. Full fine-tuning does not suppress the behaviour and may entrench it: 98.4% of spurious images remain spurious and the spurious rate rises marginally, as the shortcut is reinforced rather than removed. Critically, single-head ablation produces $p{=}0.340$ and multi-head ablation of all three identified carriers produces $p{=}0.532$,…
PDF arXiv

Papers

Recent papers

All in Research →

arXiv cs.CVOct 15
FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities
Xin Zhang, Robby T. Tan
Multi-modal semantic segmentation benefits from complementary sensors, but this benefit is fragile when some modalities are degraded or missing at test time. Many existing methods rely on dedicated fusion modules or multi-branch cross-modal encoders, and the fusion often becomes unreliable when modalities are absent, leading to substantial performance drops. We propose \textbf{FuseAdapt}, a parameter-efficient framework that performs fusion within the adaptation space of a frozen vision foundation model (VFM), exploring whether VFM priors can serve as an effective shared backbone for multi-modal segmentation. Instead of attaching a standalone fusion network, FuseAdapt assigns each modality a lightweight adaptation branch and performs \emph{adaptation-space fusion} by keeping Q/V low-rank adaptation private to each modality and routing only FFN low-rank residual activations across modalities inside the frozen backbone. The pretrained VFM remains fixed while modality-specific corrections and explicit cross-modal interaction are learned through the adaptation paths. To handle missing modalities, we further introduce \textbf{Predictive Modality Modeling (PMM)}, a direct…
PDF arXiv
arXiv cs.CVOct 15
Mixture of Multi-Sequence Feature Difference for Contrast-Enhanced MRI Synthesis
Jonghun Kim, Junmo Kwon, Hyunjin Park
Contrast-enhanced magnetic resonance imaging (CE-MRI) is crucial for accurately diagnosing and segmenting brain tumors. However, the acquisition of CE-MRI requires injecting contrast agents, which poses potential health risks and increases scan costs. Here, we propose a CE-MRI synthesis method from non-contrast sequences, reducing both risk and cost. Unlike existing methods that rely on channel-wise concatenation to fuse multi-sequence information for synthesis, we introduce a Mixture of Feature Difference (MoFD) module. The proposed module uses a dynamic weighting strategy to adaptively select and aggregate informative features from both individual source sequences and explicit differences among them. Quantitative and qualitative evaluations demonstrate that our proposed method outperforms baselines in terms of image fidelity and perceptual quality on the UCSF-PDGM and UPenn-GBM datasets. Our method successfully reconstructs tumor regions and achieves segmentation performance comparable to that obtained with real CE-MRI. Our ablation studies show that although adversarial learning improves perceptual quality, a regression-based variant of our method achieves higher structural…
PDF arXiv
arXiv cs.CVOct 15
Bridging the Markerless Tracking Validation Gap: Intraoperative X-ray Workflow Integration
Connor Daly, Salvatore Russo, Jinendra Ekanayake, Daniel S. Elson, Ferdinando Rodriguez y Baena
Markerless intraoperative tracking of spinal anatomy via RGB-D sensors is a promising alternative to invasive bone-anchored fiducials, yet no existing clinical dataset provides independent subsurface ground truth for validating these methods. We address this gap with a clinically viable pipeline that derives per-vertebra 6-DoF pose from routine intraoperative C-arm fluoroscopy, co-registered to the RGB-D camera frame through a multimodal fiducial array detectable in both imaging modalities. Critically, the pipeline requires no additional radiation, instrumentation, or deviation from standard surgical workflows, relying solely on the fluoroscopic images already acquired during routine level checks. A calibrated transform chain links the vertebral pose recovered via multi-view 2D-3D CT-to-fluoroscopy registration to the RGB-D coordinate frame, enabling direct comparison of any markerless registration estimate against a subsurface reference. Ex vivo validation against optically tracked ovine vertebrae confirms end-to-end accuracy of 2.06 +- 0.29 mm and 3.35 +- 0.93 degrees, sufficient to serve as independent ground truth for methods with substantially larger errors. We demonstrate…
PDF arXiv
arXiv cs.LGOct 15
ShiftQAT: Quantization-Aware Training for the Numerical Contract of a Fixed-Function Edge CNN Accelerator
Oguzhan Buyuksolak, Osman Erman Okman, Mehmet Gorkem Ulkar
Fixed-function CNN accelerators make edge inference fast and energy-efficient, but they execute only networks that obey a precise numerical contract. On the MAX7800x family, this contract requires power-of-two, per-tensor, symmetric scales, 8-bit activations and biases, integer-only arithmetic, and one statically configured shift per layer. We present ShiftQAT, the quantization-aware training (QAT) flow we developed for this family, which produces networks satisfying the contract by construction. It is the default training path of ai8x-training, the family's open-source training toolchain. ShiftQAT resolves residual, concatenation, and weight-sharing dependencies so that activation scales propagate statically and each layer's requantization reduces to one offline-configured shift. It also trains weights and biases on a joint power-of-two grid that fits the 8-bit bias budget. Among the compared methods, only ShiftQAT satisfies all of the contract's scaling constraints, including signed 8-bit activation storage. Even so, on ImageNet it matches the W8A8 accuracy drop of less constrained QAT methods and stays within 2.9 percentage points of FP32 at W4A8. On MAX78002 silicon,…
PDF arXiv
arXiv cs.CVOct 15
SCMerge: Spatial-Consistency-Aware Token Pair Merging for Efficient Semantic Segmentation
Junfeng Ren, Yi Gong, Yuan Zeng
Vision Transformer architectures have showcased impressive performance on semantic segmentation, yet their quadratic attention complexity hinders practical applications. Recent works have focused on merging tokens to reduce sequence length and thus improve transformer efficiency. However, existing token merging methods primarily rely on feature similarity to identify similar tokens, overlooking spatial consistency, which may introduce segmentation errors in semantic segmentation. In this work, we propose SCMerge, a training-free token merging method that jointly considers feature similarity and spatial consistency in pairing similar tokens. The pairwise tokens are then merged within the encoder, and the resulting token sequences are reconstructed onto the original patch grid after decoding. The proposed method is flexible and can be plugged into existing vision Transformer-based semantic segmentation pipelines. Extensive experiments on the benchmark datasets ADE20K, Pascal Context, and Cityscapes show that SCMerge consistently achieves a better trade-off between segmentation accuracy and efficiency than baseline methods.
PDF arXiv
arXiv cs.CVOct 15
Spurious Attention Heads in Vision Transformers: Mechanistic Identification of Annotation-Driven Bias in Medical Chest X-Ray Analysis
Joshua Muhumuza, Cornelius Kalenzi
Vision Transformers deployed in medical imaging exploit spurious correlations, yet the internal mechanisms remain unidentified and impossible to fix. We apply mechanistic interpretability to locate one such mechanism in ViT-Base 16: a single attention head (L8H10) that functions as a dedicated annotation-reader on chest X-rays, attending to corner text rather than lung pathology in 62% of 200 COVID-19 images. Causal validation via inpainting confirms annotation content rather than luminance or edge density drives the behaviour, increasing lung attention 1.52 times upon removal in affected images ($p{<}0.0001$). The finding replicates across NIH ChestX-ray14 (79%), DeiT-Base (88%), and ViT-Large 16 (80%), establishing spurious annotation-attending heads as a consistent property of ViT-family architectures across two datasets and three architectures. Full fine-tuning does not suppress the behaviour and may entrench it: 98.4% of spurious images remain spurious and the spurious rate rises marginally, as the shortcut is reinforced rather than removed. Critically, single-head ablation produces $p{=}0.340$ and multi-head ablation of all three identified carriers produces $p{=}0.532$,…
PDF arXiv

Active contributors · 90 days

Activity · 12 weeks

Coming with more sources: repos

Fields · AI · updated 21m ago

Computer vision

Perception from images and video: detection, segmentation, tracking, 3D reconstruction, medical and remote imaging.

51papers this week

All papers · 51 With code · 6

Newest Most relevant

51 papers this month

Open problems in Computer vision

View-dependent geometry features in 3D world models1 paper raises it · 1 addresses it
Entangles scene dynamics with camera motion, making state prediction harder and less generalizable.
- GeoState: Learning View-Decoupled Geometry States for 3D World Modelingraises · addresses
High-quality 3D Gaussian rendering under large viewpoint changes1 paper raises it · 1 addresses it
Feed-forward methods struggle to learn Gaussian attributes for consistent rendering across extreme viewpoints.
- InfiniSplat v2: Analyzing and Improving Feed-Forward 3DGS from Any Viewsraises · addresses
Dense 4D human correspondence without templates1 paper raises it · 1 addresses it
Existing methods fail to track surface points across articulation, deformation, and occlusion without relying on body templates.
- Dense4D: Pixel-Level 4D Human Correspondenceraises · addresses
Open-set human activity and attribute description in videos1 paper raises it · 1 addresses it
Current datasets and models simplify to closed-set or single-person tasks, limiting real-world applicability.
- Grounded Human-Attributed Description and Activity Recognition in Videosraises · addresses
Annotation-free medical image segmentation1 paper raises it · 1 addresses it
Manual annotations for tasks like retinal vessel segmentation are time-consuming and require expert knowledge.
- FPG-Seg: Annotation-free Retinal Vessel Topology Segmentationraises · addresses

Named by an AI from these papers' abstracts.