MoCKEFA: Model-Conditioned Keyframe Extraction and Fair Attribution for Video Model Explanation

Lahiru Wijayasingha

We show that a video can be represented with just a few keyframes without sacrificing a given model's belief about the video. Our method is the first of its kind that extracts these keyframes conditioned on the model's prediction, which is massively helpful for building explainable video models. We then fairly distribute the importance of the video to the model among the keyframes with game theoretically fair Shapley values. Through qualitative and quantitative experiments, we show that keyframe attribution is superior to the traditional frame attribution. Further, we conduct preliminary exploration on highlighting the important changes among the frames by looking at the differences between adjacent keyframes.