Structured Kernel Surrogate Models for Multimodal Attribution
We study the problem of cross-sample modality attribution, which involves measuring how one modality in one sample affects the learning outcome in other multimodal samples. This problem is crucial in a multimodal system because a data modality (such as a text description) may be redundant for one sample (a self-explanatory image), but the same data modality would otherwise be uniquely informative for another sample, or informative when combined with other modalities. Prior work focuses on influence estimation at the level of one sample, and it remains an open question how one should capture the interactions across both samples and modalities. To capture scenarios like the above, we propose structured kernel surrogate models, which are surrogate mappings from both samples and modalities to the prediction outcome when they are trained together. We design an algorithm, SKS, which applies kernel ridge regression with a structured, compositional kernel on the selection states of individual samples and modalities. Specifically, the kernel function multiplies kernel-based similarity scores between every pair of data modalities for one sample and a pair of two samples. Additionally, we…
Proposes structured kernel surrogate models for cross-sample modality attribution in multimodal systems.
- arXiv
- 2611.13198
- Authors
- Ziniu Zhang, Zhenshuo Zhang, Jianglin Lu, Ruoxuan Xiong, Yun Fu and 1 more
- Categories
- cs.LG · cs.AI · cs.CV