Semantic Concept Steering Breaks the Explanation Drift Loop in Continual Learning

Yehonatan Elisha, Oren Barkan, Noam Koenigstein

Catastrophic forgetting in continual learning (CL) manifests not only as accuracy degradation but also as \emph{explanation drift}: models maintain predictive accuracy while silently shifting attention from diagnostic features to spurious correlations. This work identifies and resolves a previously overlooked structural flaw in explanation-aware CL: pixel-level saliency regularization derives supervision targets from the model's own frozen saliency maps, creating a \emph{self-referential drift loop} that actively reinforces spurious attention across tasks. We introduce \textbf{C$^3$L} (Concept-Consistent Continual Learning), which breaks this loop by anchoring regularization to class-consistent semantic concepts (e.g., ``curved beak'', ``striped pattern'') discovered offline, independent of model state, providing (i) target independence from model drift, and (ii) cross-instance semantic consistency across every instance of a class. We identify that our concept-based objective, combined with approximate LRP relevance conservation, induces a \emph{crowding-out effect}: since concept and background regions compete for the relevance of the same prediction, enforcing high relevance…