Impact of color in computer and human-based object recognition under occlusion

Courtney M. King, Elissa Aminoff, Daniel D. Leeds

Partial occlusions challenge object recognition in both human and computer vision (CV), but the impact is largely dependent on the occlusion and model design. Properties relating to color are particularly important in biological recognition and frequently leveraged for camouflage, however there are still open questions about what makes them effective and how they translate to CV. This work studies partial occlusions that create camouflage, including those that blend in with target objects or with scenes, which alter object representations in distinct ways. Experiments evaluate recognition performance on both human participants and models ranging from convolutional neural networks (CNNs) to transformers, and additional analysis is performed on models to uncover trends in processing and object representations under multiple occlusion types. Novel metrics offer explainability of model behavior under occlusion: non-occlusion precision (NOP) and non-occlusion recall (NOR) measure alignment of model activations with and without occlusion, and activation in occlusion (AIO) measures the percentage of activations that occur in occluded regions. Findings show that human recognition, and…