Missingness-aware prompting for modality-missing RGBT tracking - Guyue Hu; Zhanghuan Wang; Chenglong Li; Duzhi Yuan; Bin He; Jin Tang (Journal of King Saud University: Computer and Information Sciences)
Abstract RGBT tracking has drawn great attention recently due to its ability to leverage enhancement and complementary information from the RGB and thermal infrared modalities. Nevertheless, RGBT tracking in real-world scenarios inevitably encounters heavy modality-missing challenges caused by substantial environmental factors (such as device overheating, and frame skipping). Existing methods for RGBT tracking are built upon pre-processed missingness-free datasets and suffer significant performance degradation when applied to noisy datasets with random missing modalities. In this paper, we propose a novel missingness-aware prompting framework (MAP) for modality-missing RGBT tracking. It is a lightweight prompting framework consisting of two-stage prompts focusing on compensating essential information for RGBT tracking stage-by-stage. Specifically, prototypical missingness-aware prompts (pMAP) are explored to compensate for modality-specific but instance-agnostic prototypical missing information. Contextual missingness-aware prompts (cMAP) are further designed to compensate for instance-specific detailed missing information. Extensive experiments on three large-scale datasets demonstrate the effectiveness and superiority of the proposed framework for RGBT tracking with random missing modalities.