The composition of objects and their parts, as well as the positional relationships between objects, provide a rich source of information for representation learning. Therefore, spatial cognition pretext tasks have been actively investigated in self-supervised learning. Existing studies typically start with a grid structure, and the goal of the pretext task involves predicting the absolute position index of a patch within a fixed grid. However, grid-based approaches fall short of capturing the fluid and continuous nature of the configuration of real-world objects. To overcome these limitations, we present PART, a self-supervised learning approach that leverages continuous relative transformations between off-grid patches. By modeling how parts relate to each other in continuous space, PART learns the relative composition of an image, the relative position of an off-grid structure that is less constrained by absolute appearance and can remain consistent under changes such as partial visibility or style changes. PART outperforms grid-based methods such as MAE and DropPos for tasks requiring accurate spatial understanding, such as object detection and time series prediction, and maintains competitive performance for global classification tasks. Freed from grid constraints, PART opens a new path for universal self-supervised pre-training across diverse data types, from images to EEG signals, with potential for medical images, video, and audio.
- † University of Amsterdam
