- Learning Under Distribution Mismatch and Model Specification Mismatch (arXiv)
Author : Saeed Masiha, Amin Gohali, Mohammad Hossein Yassai, Mohammad Reza Aleph
Abstract: We study learning algorithms when there is a discrepancy between the distributions of the training and test datasets of the learning algorithm. The impact of this discrepancy on generalization error and model misspecification is quantified. In addition, we provide the relationship between generalized error and rate-distortion theory. This allows the bounds of rate-distortion theory to be used to derive new bounds for the generalization error, and vice versa. In particular, the rate-distortion-based bound is a stricter improvement than the previous one by Xu and Raginsky, even in the absence of mismatch. We also show how an “auxiliary loss function” can be used to obtain an upper bound on the generalized error.
2. MixMOOD: A Systematic Approach to Class Distribution Discrepancies in Semi-Supervised Learning Using Deep Dataset Dissimilarity Measures (arXiv)
Author : Saul Calderon-Ramires, Luis Oala, Jordina Trento-Valena, Shenxiang Yang, Almahan Momeni, Wojciech Samek, Miguel A. Molina-Cabello
Abstract: In this work, we propose MixMOOD, a systematic approach to using MixMatch to mitigate the effects of class distribution mismatch in semi-supervised deep learning (SSDL). This study he divided into two components: (i) an extensive out-of-distribution (OOD) ablation testbed for SSDL, and (ii) a quantitative unlabeled dataset selection heuristic called MixMOOD. The first part analyzes the sensitivity of MixMatch accuracy under 90 different distribution mismatch scenarios across three multiclass classification tasks. These are designed to systematically understand how OOD’s unlabeled data impacts her MixMatch performance. In the second part, we propose an efficient and effective method called deep dataset dissimilarity measures (DeDiMs) for comparing labeled and unlabeled datasets. The proposed DeDiM can be quickly evaluated and modeled in a perception-agnostic manner. They use a generic Wide-ResNet feature space and can be applied before training. Our test results reveal that the assumed semantic similarity between labeled and unlabeled data is not a good heuristic for choosing unlabeled data. In contrast, since there is a strong correlation between MixMatch accuracy and the proposed DeDiM, we can pre-quantitatively rank different unlabeled datasets according to their expected MixMatch accuracy. This is what is called MixMOOD. Furthermore, we argue that the MixMOOD approach helps standardize the evaluation of various semi-supervised learning techniques under real-world scenarios involving out-of-distribution data.
