Zero-shot visual reasoning through probabilistic analogical mapping

The ability to discern abstract similarities between superficially different visual inputs is a cornerstone of human reasoning. Children as young as four can answer questions like, "If a tree had a knee, where would it be?", demonstrating an innate understanding of spatial relationships across diverse object categories. Cognitive science has explored the computational principles underlying analogical reasoning, focusing on structured representations, binding between entities and relations, and mapping mechanisms. However, these models often rely on manually constructed representations, lacking an explanation of how these representations are derived from real-world perceptual data, particularly for visual analogies. Unlike linguistic analogies where relations are explicitly stated, visual analogies necessitate a mechanism to extract relations from non-relational inputs like image pixels. Deep learning approaches have attempted to solve visual analogy problems directly from pixel-level inputs, but these methods often require extensive training data (sometimes over a million examples) and show limited generalization to novel content. These models typically contrast with human reasoning, which often performs zero-shot analogical reasoning—solving problems without prior training, only with general task instructions and possibly a practice problem. This contrasts with the data-intensive methods used in deep learning approaches. This paper presents visiPAM, a model that combines the strengths of learned representations derived from pixel-level or 3D point-cloud inputs with a reasoning mechanism inspired by cognitive models of analogy to address the challenges of zero-shot visual reasoning.

Existing research in visual analogy has followed two distinct paths. Cognitive science models have focused on the mechanisms of human analogical reasoning, emphasizing structured representations and similarity-based mapping. However, these models typically rely on hand-crafted representations that don't account for how such representations arise from raw perceptual input. In contrast, deep learning approaches have directly tackled visual analogy tasks using end-to-end training from pixel inputs. While effective within their training domain, these methods typically struggle with generalization to unseen data due to their reliance on massive datasets and lack of structured representations. Recent attempts to improve this generalization have focused on learning from fewer examples or improved out-of-distribution generalization, but they still require at least some direct training on the task, unlike humans.

VisiPAM, the proposed model, addresses the limitations of previous approaches by combining learned representations from deep learning with a reasoning mechanism inspired by cognitive models. VisiPAM's architecture consists of two core components: a vision module and a reasoning module. The vision module extracts structured visual representations from either 2D images or 3D point clouds. For 2D images, it uses iBOT, a self-supervised vision transformer, to extract node attributes capturing visual appearance. For 3D point clouds, it uses a Dynamic Graph Convolutional Neural Network (DGCNN) trained on a part segmentation task. In both cases, the output is an attributed graph representation where nodes represent object parts, and edges represent spatial relations. The reasoning module employs Probabilistic Analogical Mapping (PAM), a Bayesian inference method that identifies correspondences between the source and target graphs based on node and edge similarities. PAM uses a graduated assignment algorithm to find the optimal mapping, balancing node and edge similarities and favoring isomorphic mappings. This involves a weighted combination of node and edge similarity scores, with a parameter α controlling the balance. The model's performance was evaluated on two tasks: a part-matching task with 2D images (Pascal Part Matching dataset) and a novel task involving 3D objects, comparing visiPAM's performance to human subjects. For the 3D task, point clouds were generated from 3D models and analyzed with a clustering algorithm (KMeans++) to define nodes. Edge attributes were computed using 3D spatial relations. In both cases, the model's performance is analyzed based on mapping accuracy, with comparisons made to other state-of-the-art deep learning models and human performance. Ablation studies were conducted to assess the contributions of node and edge similarities to overall performance.

VisiPAM, without any direct training on the part-matching task, significantly outperformed the state-of-the-art Structured Set Matching Network (SSMN) on a part-matching task using 2D images. This represented a 30% relative reduction in the error rate. VisiPAM demonstrated comparable performance across various conditions, including within-category and between-category comparisons. Ablation studies showed that both node and edge similarity were crucial for visiPAM's performance. Furthermore, the model's performance was robust to low-level image manipulations, like horizontal reflections. However, visiPAM showed limitations in handling certain object categories, particularly planes, due to the limitations of 2D spatial representations. In the 3D object mapping task, visiPAM's performance closely matched the pattern of human behavior. Both the model and human subjects showed greater variability in mappings between objects from different superordinate categories compared to those from the same category. VisiPAM's predictions were highly correlated with human responses at the item level (r=0.70), indicating strong alignment with human analogical mapping patterns. Ablation studies also demonstrated that both node and edge similarities were crucial for predicting human responses. The model displayed impressive successes in mapping across significant visual differences in images (background, lighting, pose, and visual appearance), both within and between object categories. However, errors often involved confusion of corresponding lateralized parts, highlighting limitations in representing complex spatial relationships, particularly in 3D object mapping.

The results demonstrate that visiPAM effectively combines the representational power of deep learning with the similarity-based reasoning operations of human cognition to achieve zero-shot analogical mapping. VisiPAM's success in outperforming a state-of-the-art deep learning model without any direct training on the mapping task highlights the importance of incorporating structured representations and cognitive principles into AI systems. The close alignment between visiPAM's performance and human behavior on the 3D object mapping task provides further evidence for the validity of its underlying approach. The findings suggest that the object-level similarity, often considered a limitation in human reasoning, may serve as a useful constraint in complex real-world analogy-making. The integration of node and edge similarity plays a pivotal role, echoing human analogical reasoning’s sensitivity to both entity and relational similarity. The superior performance of visiPAM, even compared to recent deep learning models that achieve greater out-of-distribution generalization, underscore the efficacy of zero-shot learning combined with structured representations.

VisiPAM offers a novel approach to zero-shot visual reasoning, successfully integrating learned visual representations with a cognitively inspired reasoning mechanism. Future work could focus on deriving 3D representations directly from 2D inputs, incorporating non-visual information, integrating top-down processing for improved representation learning, and extending the model to handle more complex scenarios such as multiple analogies or schema induction. The synergy between advanced visual representations and human-like reasoning holds significant potential for advancing AI capabilities in visual reasoning.

One limitation is the reliance on pre-trained models (iBOT and DGCNN) for visual representation learning. The performance of visiPAM is directly affected by the quality of these representations. Another limitation is the lack of a mechanism to derive 3D representations from 2D inputs, a task that humans readily perform. While topological information improved performance, a more robust solution is needed. The model’s occasional confusion of lateralized parts also indicates a need for more sophisticated spatial relationship modeling. Finally, the model’s current formulation is purely bottom-up. Incorporating top-down feedback mechanisms into the architecture could enhance its performance and generalization capabilities.