Papers

Field

Top tools here

Open problems in Adversarial training

Robust adversarial detection with theoretical guarantees3 papers raise it · 3 address it
Current detectors rely on heuristics and fail under adaptive attacks, leaving models vulnerable to adversarial inputs.
Emergent misalignment in multimodal models1 paper raises it · 1 addresses it
Fine-tuning on narrow tasks can induce broadly harmful behaviors in vision-language models, but its mechanisms and mitigation are poorly understood.
- Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignmentraises · addresses
Safe and high-quality generation trade-off1 paper raises it · 1 addresses it
Restricting training data for safety reduces model quality, while post-hoc unlearning risks catastrophic forgetting.
- Pristine Modelsraises · addresses
Robustness of decision models under adversarial manipulation1 paper raises it · 1 addresses it
Decision models are increasingly used for automated tasks, but their reliability under attacks is untested and unstandardized.
- DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Visionraises · addresses
Watermark removal resilience1 paper raises it · 2 address it
Invisible watermarks for ownership verification are vulnerable to simple, training-free attacks that preserve image quality.
- BrushOff: Stroke-Based Re-rendering as a Watermark Removal Attack on Paintingraises · addresses
- SkillMarker: A KGW-Based Watermarking Method for Agent Skill Ownership Verificationaddresses

Named by an AI from these papers' abstracts.

Open problems

Recently funded

Topics in this field

deepfake detection · forensic cues · image forgery107 papers21 this week
adversarial training · adversarial attacks · adversarial images107 papers14 this week
watermarking schemes · watermarking · based watermarking76 papers11 this week

Topics in this field

Active contributors · 90 days

Active contributors

Activity · 12 weeks

Coming with more sources: repos

Activity

Recent papers

All in Research →

arXiv cs.CVOct 4
VisualErase: Dual-Branch Visual Trajectory Redirection for Robust Concept Erasure in Text-to-Image Diffusion Models
Qianlong Xiang, Miao Zhang, Kun Wang, Yupeng Hu, Junhui Hou +1
Concept erasure is essential for the safe deployment of text-to-image diffusion models, as they may reproduce harmful, copyrighted, or privacy-sensitive content learned from unconstrained large-scale data. Existing methods typically erase unwanted concepts while preserving general generation capability by redirecting target-related text-to-image mappings. However, recent studies show that erased models may still retain visual generative trajectories of target concepts, leaving them vulnerable to adversarial recovery attacks and revealing a fundamental gap between redirecting text-to-image mappings and truly removing visual knowledge. To bridge this gap, we propose VisualErase, a new paradigm that redirects concept-bearing visual generative trajectories toward explicitly defined concept-removed outcomes. To enable this redirection, we use structure-preserving image editing to construct content-aligned, concept-removed counterparts for source images, providing explicit visual endpoints that retain non-target content. We then derive a denoising target from each source-to-counterpart pair and use a dual-branch redirection loss to align both text-conditioned and unconditional…
PDF arXiv
arXiv cs.CROct 4
Invisible Ink, Visible Lies: How Production Watermarking Causes LLMs to Hallucinate
Haocheng Ye, Aoting Hu, Xinwei Zhang, Xunzhu Tang, Shuchao Pang +1
Text watermarking helps identify AI-generated content, but its effect on factual reliability remains underexplored. In this paper, we study watermarking hallucination: factual errors induced or amplified by watermarking even when the required evidence is present in the context and the unwatermarked model can answer correctly. Using a controlled retrieval-augmented generation setting, we compare unwatermarked and watermarked generations under the same context, query, and decoding configuration, and quantify their factual accuracy decrease. Across six representative watermarking methods, including KGW, SWEET, DiPmark, GumbelSoft, Gumbel-Max, and SynthID watermarking, we consistently observe watermark-induced hallucination. Watermarked outputs can remain fluent while introducing factual errors. We attribute this failure mode to two mechanisms: (1) token perturbations in the current-step arising from method-specific reweighting or keyed sampling, and (2) prefix-induced attention drift, which accumulates through autoregressive decoding and weakens later attention to the factual context. Motivated by this analysis, we propose two plug-in interventions at the token and attention levels…
PDF arXiv
arXiv cs.CVOct 4
Visual-Invariance-Augmented Feature Optimal Alignment for Transferable Adversarial Attacks against Closed-Source MLLMs
Xiaojun Jia, Simeng Qin, Yiming Li, Jie Liao, Sensen Gao +3
Multimodal large language models (MLLMs) remain vulnerable to transferable adversarial examples, especially in black-box settings where only open-source surrogate models are accessible. Existing targeted transfer attacks mainly align adversarial and target samples using global image-level features, such as encoder [CLS] embeddings. However, such coarse alignment insufficiently exploits patch-level visual structures, limiting transferability across heterogeneous closed-source MLLMs. We propose IAU-FOA, a visual-invariance-augmented feature optimal alignment attack with adaptive unbalanced transport, to improve targeted transferability against closed-source MLLMs. IAU-FOA aligns adversarial and target samples at both global and local levels: a cosine-based objective narrows their global semantic gap, while patch tokens are clustered into compact local patterns and matched through optimal transport for fine-grained feature alignment. Balanced optimal transport enforces fixed marginal masses even for local clusters without reliable counterparts, potentially introducing misleading alignment gradients. We therefore introduce confidence-adaptive unbalanced transport to relax these…
PDF arXiv
arXiv cs.LGOct 3
Pareto-Improving Adversarial Attacks with Primal-Dual Regularization
Yang Dai, Longfei Zhang, Wei Tao, Li Shen, Jincai Huang +1
Transferable adversarial attacks are arguably the most practical black-box threat model. Under the same perturbation budget, stronger transfer attacks attain higher attack success rate (ASR), yet their imperceptibility also tends to degrade. Under such a fixed-budget protocol, transferability and imperceptibility therefore appear to trade off against each other. We argue that this conflict is an artifact of fixed-budget evaluation, not an intrinsic trade-off. When attacks are compared on the ASR--imperceptibility Pareto frontier obtained by sweeping $ε$, stronger transfer attacks already attain better imperceptibility at matched ASR than weaker ones. To exploit this latent advantage, we introduce the stealthy transfer attack ST, a plug-in primal-dual wrapper that adds an $L_\infty$ saturation regularizer to the standard constrained objective and resolves it through a two-step primal-dual update: a projected primal step on the perturbation coupled with an $L_1$-ball projection on a dual variable that absorbs the regularizer through Fenchel duality, requiring no auxiliary models or handcrafted perceptual priors. Empirically, ST extends the Pareto frontier across different base…
PDF arXiv
arXiv cs.CROct 3
GrayShield: Bit-Level Sanitization for Transformer Model Supply-Chain Security
Armstrong Foundjem, Tsung-Hsien Chuang, Foutse Khomh, Mohamed Amine Merzouk
Transformer models such as BERT and Vision Transformer~(ViT) achieve strong performance via densely parameterized attention backbones. However, the least significant bits~(LSBs) of their 32-bit floating-point weights can be abused as covert channels to conceal malicious payloads, posing a serious threat to the AI model supply chain. We propose \GS (\GSabbr), a lightweight, post-training, zero-data sanitization method that completely replaces the declared mantissa-LSB channel with a Gray-code-guided low-transition sequence. Complete payload-independent overwrite, whether keyed or public, makes the sanitized target bits independent of the embedded payload and gives that declared channel zero capacity. Gray coding supplies overwrite structure, while a keyed per-tensor phase supplies pattern diversity. Benchmarked against seven post-training defenses on four Transformer model presets and two real-world malware payloads, \GSabbr maintains sub-$1\%$ accuracy impact and achieves $49.96\pm0.66$ percentage-point Recovery Reduction (RR) under five implemented attacker variants. Because pre-defense recovery is effectively $100\%$, RR near 50 percentage points corresponds to…
PDF arXiv
arXiv cs.CROct 3
Language Model Fingerprinting Requires Rethinking Watermark Teachers
Jeongyeon Hwang, Anshul Nasery, Sewoong Oh, Jungseul Ok
LLM fingerprinting via watermark distillation embeds a statistical watermark signal into model weights, enabling model owners to identify their models behind black-box APIs. Revisiting a recent protocol, we find that its utility evaluation understates text quality degradation in open-ended generation, favoring overly strong watermark teachers. Weakening the watermark improves text quality but sacrifices detectability. To move beyond this trade-off, we rethink whether text watermarks designed for verifying generated text are suitable distillation teachers for model fingerprinting. Such watermarks are typically designed to remain detectable from an individual output, limiting how sparse the watermark signal can be. In contrast, fingerprint verification can aggregate signal across queries, making sparser watermark signals viable. This raises a key question: where should the sparse signal be placed? We analyze signal placement through token surprisal and show that, even at comparable watermark strength, different placements can target tokens with different plausibility under the base model. This motivates near-tie restriction, which uses top-1-relative logit gaps to restrict the…
PDF arXiv

Papers

Topics in this field

deepfake detection · forensic cues · image forgery107 papers21 this week
adversarial training · adversarial attacks · adversarial images107 papers14 this week
watermarking schemes · watermarking · based watermarking76 papers11 this week

Active contributors · 90 days

Recent papers

All in Research →

arXiv cs.CVOct 4
VisualErase: Dual-Branch Visual Trajectory Redirection for Robust Concept Erasure in Text-to-Image Diffusion Models
Qianlong Xiang, Miao Zhang, Kun Wang, Yupeng Hu, Junhui Hou +1
Concept erasure is essential for the safe deployment of text-to-image diffusion models, as they may reproduce harmful, copyrighted, or privacy-sensitive content learned from unconstrained large-scale data. Existing methods typically erase unwanted concepts while preserving general generation capability by redirecting target-related text-to-image mappings. However, recent studies show that erased models may still retain visual generative trajectories of target concepts, leaving them vulnerable to adversarial recovery attacks and revealing a fundamental gap between redirecting text-to-image mappings and truly removing visual knowledge. To bridge this gap, we propose VisualErase, a new paradigm that redirects concept-bearing visual generative trajectories toward explicitly defined concept-removed outcomes. To enable this redirection, we use structure-preserving image editing to construct content-aligned, concept-removed counterparts for source images, providing explicit visual endpoints that retain non-target content. We then derive a denoising target from each source-to-counterpart pair and use a dual-branch redirection loss to align both text-conditioned and unconditional…
PDF arXiv
arXiv cs.CROct 4
Invisible Ink, Visible Lies: How Production Watermarking Causes LLMs to Hallucinate
Haocheng Ye, Aoting Hu, Xinwei Zhang, Xunzhu Tang, Shuchao Pang +1
Text watermarking helps identify AI-generated content, but its effect on factual reliability remains underexplored. In this paper, we study watermarking hallucination: factual errors induced or amplified by watermarking even when the required evidence is present in the context and the unwatermarked model can answer correctly. Using a controlled retrieval-augmented generation setting, we compare unwatermarked and watermarked generations under the same context, query, and decoding configuration, and quantify their factual accuracy decrease. Across six representative watermarking methods, including KGW, SWEET, DiPmark, GumbelSoft, Gumbel-Max, and SynthID watermarking, we consistently observe watermark-induced hallucination. Watermarked outputs can remain fluent while introducing factual errors. We attribute this failure mode to two mechanisms: (1) token perturbations in the current-step arising from method-specific reweighting or keyed sampling, and (2) prefix-induced attention drift, which accumulates through autoregressive decoding and weakens later attention to the factual context. Motivated by this analysis, we propose two plug-in interventions at the token and attention levels…
PDF arXiv
arXiv cs.CVOct 4
Visual-Invariance-Augmented Feature Optimal Alignment for Transferable Adversarial Attacks against Closed-Source MLLMs
Xiaojun Jia, Simeng Qin, Yiming Li, Jie Liao, Sensen Gao +3
Multimodal large language models (MLLMs) remain vulnerable to transferable adversarial examples, especially in black-box settings where only open-source surrogate models are accessible. Existing targeted transfer attacks mainly align adversarial and target samples using global image-level features, such as encoder [CLS] embeddings. However, such coarse alignment insufficiently exploits patch-level visual structures, limiting transferability across heterogeneous closed-source MLLMs. We propose IAU-FOA, a visual-invariance-augmented feature optimal alignment attack with adaptive unbalanced transport, to improve targeted transferability against closed-source MLLMs. IAU-FOA aligns adversarial and target samples at both global and local levels: a cosine-based objective narrows their global semantic gap, while patch tokens are clustered into compact local patterns and matched through optimal transport for fine-grained feature alignment. Balanced optimal transport enforces fixed marginal masses even for local clusters without reliable counterparts, potentially introducing misleading alignment gradients. We therefore introduce confidence-adaptive unbalanced transport to relax these…
PDF arXiv
arXiv cs.LGOct 3
Pareto-Improving Adversarial Attacks with Primal-Dual Regularization
Yang Dai, Longfei Zhang, Wei Tao, Li Shen, Jincai Huang +1
Transferable adversarial attacks are arguably the most practical black-box threat model. Under the same perturbation budget, stronger transfer attacks attain higher attack success rate (ASR), yet their imperceptibility also tends to degrade. Under such a fixed-budget protocol, transferability and imperceptibility therefore appear to trade off against each other. We argue that this conflict is an artifact of fixed-budget evaluation, not an intrinsic trade-off. When attacks are compared on the ASR--imperceptibility Pareto frontier obtained by sweeping $ε$, stronger transfer attacks already attain better imperceptibility at matched ASR than weaker ones. To exploit this latent advantage, we introduce the stealthy transfer attack ST, a plug-in primal-dual wrapper that adds an $L_\infty$ saturation regularizer to the standard constrained objective and resolves it through a two-step primal-dual update: a projected primal step on the perturbation coupled with an $L_1$-ball projection on a dual variable that absorbs the regularizer through Fenchel duality, requiring no auxiliary models or handcrafted perceptual priors. Empirically, ST extends the Pareto frontier across different base…
PDF arXiv
arXiv cs.CROct 3
GrayShield: Bit-Level Sanitization for Transformer Model Supply-Chain Security
Armstrong Foundjem, Tsung-Hsien Chuang, Foutse Khomh, Mohamed Amine Merzouk
Transformer models such as BERT and Vision Transformer~(ViT) achieve strong performance via densely parameterized attention backbones. However, the least significant bits~(LSBs) of their 32-bit floating-point weights can be abused as covert channels to conceal malicious payloads, posing a serious threat to the AI model supply chain. We propose \GS (\GSabbr), a lightweight, post-training, zero-data sanitization method that completely replaces the declared mantissa-LSB channel with a Gray-code-guided low-transition sequence. Complete payload-independent overwrite, whether keyed or public, makes the sanitized target bits independent of the embedded payload and gives that declared channel zero capacity. Gray coding supplies overwrite structure, while a keyed per-tensor phase supplies pattern diversity. Benchmarked against seven post-training defenses on four Transformer model presets and two real-world malware payloads, \GSabbr maintains sub-$1\%$ accuracy impact and achieves $49.96\pm0.66$ percentage-point Recovery Reduction (RR) under five implemented attacker variants. Because pre-defense recovery is effectively $100\%$, RR near 50 percentage points corresponds to…
PDF arXiv
arXiv cs.CROct 3
Language Model Fingerprinting Requires Rethinking Watermark Teachers
Jeongyeon Hwang, Anshul Nasery, Sewoong Oh, Jungseul Ok
LLM fingerprinting via watermark distillation embeds a statistical watermark signal into model weights, enabling model owners to identify their models behind black-box APIs. Revisiting a recent protocol, we find that its utility evaluation understates text quality degradation in open-ended generation, favoring overly strong watermark teachers. Weakening the watermark improves text quality but sacrifices detectability. To move beyond this trade-off, we rethink whether text watermarks designed for verifying generated text are suitable distillation teachers for model fingerprinting. Such watermarks are typically designed to remain detectable from an individual output, limiting how sparse the watermark signal can be. In contrast, fingerprint verification can aggregate signal across queries, making sparser watermark signals viable. This raises a key question: where should the sparse signal be placed? We analyze signal placement through token surprisal and show that, even at comparable watermark strength, different placements can target tokens with different plausibility under the base model. This motivates near-tie restriction, which uses top-1-relative logit gaps to restrict the…
PDF arXiv

Activity · 12 weeks

Coming with more sources: repos

Open problems in Adversarial training

Robust adversarial detection with theoretical guarantees3 papers raise it · 3 address it
Current detectors rely on heuristics and fail under adaptive attacks, leaving models vulnerable to adversarial inputs.
Emergent misalignment in multimodal models1 paper raises it · 1 addresses it
Fine-tuning on narrow tasks can induce broadly harmful behaviors in vision-language models, but its mechanisms and mitigation are poorly understood.
- Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignmentraises · addresses
Safe and high-quality generation trade-off1 paper raises it · 1 addresses it
Restricting training data for safety reduces model quality, while post-hoc unlearning risks catastrophic forgetting.
- Pristine Modelsraises · addresses
Robustness of decision models under adversarial manipulation1 paper raises it · 1 addresses it
Decision models are increasingly used for automated tasks, but their reliability under attacks is untested and unstandardized.
- DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Visionraises · addresses
Watermark removal resilience1 paper raises it · 2 address it
Invisible watermarks for ownership verification are vulnerable to simple, training-free attacks that preserve image quality.
- BrushOff: Stroke-Based Re-rendering as a Watermark Removal Attack on Paintingraises · addresses
- SkillMarker: A KGW-Based Watermarking Method for Agent Skill Ownership Verificationaddresses

Named by an AI from these papers' abstracts.

Fields · AI · updated 46m ago

Adversarial training

Found by the topic model: adversarial training, deepfake detection, watermarking schemes.

46papers this week−22.0% vs last week

All papers · 290 With code · 3

Newest Most relevant

273 papers this month