The role of semantics in the perceptual organization of shape

Successful interaction with objects is crucial for survival, requiring the inference of object properties (material, usage, dangerousness) largely based on past experiences with similar objects. Object shape is a primary cue for object recognition and concept learning, playing a major role in generalization across object classes. This study focuses on the ability to identify point-to-point correspondences between objects, a key aspect of shape understanding. Previous research demonstrated that perceived correspondence is robust against viewpoint changes and transformations, even at the level of point-to-point correspondence. A heuristic model based on contour curvature successfully predicted human responses in some cases. However, this approach fails to explain correspondences between objects with vastly different shapes, suggesting the involvement of semantic information. The hypothesis is that humans solve correspondence tasks by combining shape and semantic information, segmenting objects into recognizable parts (legs, wings, tails) that can be matched across objects, and using a heuristic to interpolate between these correspondences. This would allow for correspondence even when contour shapes differ significantly. The study aims to demonstrate how correspondence between very different objects is established by evaluating similarity between semantic parts, combining perceptual organization and cognitive processes.

Prior studies explored point-to-point correspondence between objects undergoing rigid and non-rigid transformations, demonstrating robustness against viewpoint changes. A curvature-based heuristic model effectively predicted human performance in these scenarios. However, these studies primarily focused on transformations within similar object classes, not the wide disparities in shape found across diverse object categories (e.g., butterflies and owls). Existing models based purely on geometric features, such as curvature, fail to account for the human ability to identify correspondences between objects with substantially different shapes but similar semantic part structures (e.g., the trunk of an elephant and the snout of an anteater). This suggests the involvement of higher-level cognitive processes, particularly semantic understanding, which is under-explored in the existing literature on shape correspondence.

The study used three experiments to investigate the role of semantics in establishing point-to-point correspondences. Experiment 1 examined correspondence between object pairs (e.g., elephant-anteater) with different geometries but similar part organizations. Fifteen participants were presented with six pairs of 2D shapes. For each base shape, 50 probe points were sampled, and participants placed corresponding points on the test shape. Agreement between participants, preserved ordering of locations, and correspondence on similar semantic parts were analyzed. Experiment 2 explored correspondence between objects (e.g., swan-squirrel) with identical geometry but different semantic part organizations (ambiguous figures). Fifteen participants completed the same task with five ambiguous figure pairs and two baseline conditions (identical shape and label). The analysis followed the same structure as Experiment 1, examining participant agreement, ordering preservation, and the influence of semantic interpretation. Experiment 3 aimed to obtain semantic labels for the shapes used in Experiments 1 and 2. In Experiment 3A, 12 participants labelled parts of each shape from a list of 16 common labels (head, body, legs, etc.). In Experiment 3B, 9 participants matched the labels across the corresponding shapes. This data formed the basis for a semantic organization model to predict participant responses in Experiments 1 and 2. The model was compared to alternative models: a uniform sampling model and a curvature-based model, both utilizing dynamic time warping to align surprisal profiles. A combined model integrated semantic and curvature information was also tested.

Experiment 1 demonstrated high inter-participant agreement in establishing correspondences across different shapes with similar part organizations, significantly exceeding random performance. Order preservation was generally high, with correspondences frequently aligning with similar semantic parts. The Butterfly-Owl pair showed lower agreement due to ambiguous 3D orientation. Experiment 2 revealed that even with identical shapes, correspondences differed based on semantic part organization, indicating the dominance of semantics over purely geometric features. Lower congruency compared to Experiment 1 was likely due to less clear part organization and correspondence in the ambiguous figures. Ambiguous orientations also affected congruency in the Parrot-Goose and Duck-Rabbit pairs. Experiment 3 provided semantic labels for the shapes, allowing the creation of a semantic organization model. This model predicted human responses as well as other human responses did for most pairs in both Experiments 1 and 2, outperforming both uniform sampling and curvature-based models. The combined semantic and curvature-based model did not significantly improve predictions over the semantic-only model. Statistical analyses (t-tests and Bayes factors) supported the model's superiority.

The findings support the hypothesis that humans flexibly utilize both perceptual (shape) and cognitive (semantic) processes to establish point-to-point correspondences. For unfamiliar shapes, salient geometric features guide correspondence; for familiar shapes with similar part structures, semantic information takes precedence. The semantic organization model effectively captures this flexible strategy, demonstrating the significant role of semantic organization in shaping perceptual judgments of correspondence. The model's superior performance compared to geometric-only models highlights the limitations of purely bottom-up approaches and underscores the integration of top-down cognitive processes in shape perception. The results have implications for understanding object constancy, similarity judgments, and the generalization across object classes.

This study demonstrates the crucial role of semantic part organization in establishing point-to-point correspondences between objects, particularly those with significantly different shapes. A novel semantic organization model accurately predicts human responses, outperforming geometry-based models. Future research should explore the integration of this model with machine learning techniques for improved shape morphing and more human-like object comparison. Investigating the interplay between perceptual and cognitive processes in neural networks is another promising direction.

The accuracy of the model depends on the quality and detail of semantic part information, which might vary between participants and stimuli. The model does not fully account for ambiguities in shape interpretation or inter-individual differences. Future work could address these limitations by incorporating probabilistic predictions based on semantic labeling frequencies and extending the model to 3D shapes.