Jeffrey Beng-Hee Ho
Department of Psychology
Carnegie Mellon University
Pittsburgh, PA 15213-3890
ho+@cmu.edu
Marlene Behrmann
Department of Psychology
Carnegie Mellon University
Pittsburgh, PA 15213-3890
mb9h@crab.psy.cmu.edu
David C. Plaut
Department of Psychology
Carnegie Mellon University, and
Center for the Neural Basis of Cognition
Pittsburgh, PA 15213-3890
plaut@cmu.edu
In Proceedings of the 17th Annual Conference of the Cognitive Science Society, pages 148-153. Hillsdale, NJ: Lawrence Erlbaum Associates.
A central question in understanding neglect is what constitutes ``left''---that is, With respect to what frame of reference is the left side defined? Possibilities include viewer-centered frames (e.g., aligned with the retina, head, or trunk), environment-centered frames (e.g., aligned with the room, table, or page), and object-centered frames (i.e., aligned with depicted objects). Under most viewing conditions, these frames are all aligned, so there is no way to evaluate which frame determines the nature of neglect behavior.
In fact, recent evidence suggests that neglect behavior is sensitive to spatial information defined simultaneously with respect to multiple reference frames. When viewer-centered and object-centered frames are deconfounded by rotating the stimulus or the viewer, patients continue to exhibit viewer-centered neglect but also fail to report information on the left of the object even though this information is located to the right of midline of the viewer and/or the environment (Behrmann & Moscovitch, 1994, Driver & Halligan, 1991, Young, Hellawell, & Welch, 1992). A compelling example of this object-centered neglect is the case of NG, a patient with right-sided neglect, who failed to read the rightmost letters of a word even when the word was presented vertically or in mirror-reverse format (Caramazza & Hillis, 1990). Object-centered neglect can also be demonstrated under fixed viewing conditions. When viewing an equilateral triangle with a gap on one side, patients with left-sided neglect fail to detect the gap more often when they are biased to see the triangle as pointing in a direction that places the gap on the left of the perceived major axis (Driver, Baylis, Goodrich, & Rafal, 1994). These data support the view that the spatial positions of parts of an object are coded with respect to a reference frame aligned with the principle axis of the object itself (Marr, 1982, Marr & Nishihara, 1978), and that visual attention is allocated, at least in part, relative to this frame.
Object-centered effects in neglect may manifest in more complicated ways in a copying task, in which a target stimulus, often a daisy or a clock, is presented upright in the center of a blank piece of paper. Patients with neglect often produce an adequate representation of the right side of the figure while leaving out significant features on the left. For example, the left drawing in Figure 1 shows the performance of a neglect patient, JM, in copying an upright daisy, in which the leftmost petals were omitted. Again, though, the standard copying task confounds the influences of reference frames centered on the viewer, the environment, and the object. If parts are located with respect to an object-centered frame, then, when copying a daisy rotated from the upright, patients should continue to neglect to draw parts on the left defined intrinsically with respect to the object itself. As is evident in Figure 1, patient JM still omits features on the object-centered left of the daisy when copying misoriented versions of the target daisy.
Figure 1:
Neglect patient JM's copying of a daisy presented in different
orientations.
Interestingly, object-centered neglect in copying may occur not only for a single object but also when the figure to be copied contains multiple items. In this case, patients may omit features on the left of an object while still including features on the right of another object that is further to the left on the page (Gainotti, Messerli, & Tissot, 1972, Marshall & Halligan, 1993). Moreover, the object-centered deficit may appear even within subparts of a single, complex object. When patient PP (Driver & Halligan, 1991) was required to copy a single wheel, presented on the left or right of a page, she omitted spokes on the left of the wheel. When two wheels were presented, one on the left and one on the right, she omitted spokes only on the left of the left wheel. When the two wheels appeared as parts of a larger object (a bicycle), PP omitted the left wheel entirely, retaining only the right one.
These findings make sense if the representation of an object has a hierarchical structure in which its parts are in themselves objects at a smaller spatial scale, and which decompose further into their own parts at an even smaller scale (Marr & Nishihara, 1978, Palmer, 1977). During the copying of a complex figure, a reference frame aligned with a part of the object serves as the context frame for locating and drawing its subparts. Thus, the object-centered frame is not fixed throughout the task; rather, objects are recursively decomposed and dynamically assigned to roles as objects and parts depending on the current relevant level of the hierarchy (see Hinton, 1990). Accounting for the copying performance of neglect patients (and of normal subjects) is complicated because, at one point in time, the context frame may represent the spatial coordinates for copying a particular part, whereas at a second point in time, this same part may itself define the context frame for the copying of its own subparts.
The goal of this paper is to examine how hierarchical object representations might interact with spatial reference frames to explain the performance of patients who show neglect both with respect to multiple frames of reference and at multiple levels of the object hierarchy. First, we examine whether, as suggested above, the performance of normal subjects in copying misoriented versions of a daisy is mediated by a hierarchical representation of the daisy. We then implement this process as a conventional tree-traversal algorithm over a hierarchical data structure representing the daisy. During the traversal, the position of each component is maintained relative to both the local object-centered frame and the global viewer-centered frame. By imposing a spatially defined lesion, analogous to the deficit hypothesized to underlie the attentional impairment in patients with right-parietal damage, we demonstrate how neglect can arise with respect to both the viewer- and object-centered reference frames even when objects are misaligned from their canonical orientation.
To verify that this hierarchical object representation adequately captures drawing performance, we had 20 normal subjects generate 3 copies of a daisy presented in each of 4 orientations (upright, 90 deg rotation to the left or right, and inverted). We tracked the order of strokes used by the subjects. Drawing performance was considered to obey the hierarchical representation if the order in which the components were drawn followed a depth-first traversal order through the hierarchy (ignoring the order among subparts). In other words, once a stroke within a particular subtree is drawn, all of its components and subcomponents must be drawn before a stroke within another subtree at the same level is drawn. Any stroke that did not adhere to this rule was counted as a violation of the hierarchy. Across all subjects and drawing conditions, the mean number of hierarchy violations was 1.3 (SD 0.84), and was not significantly affected by the orientation of the daisy (F<1). This number of violations is significantly different from the mean obtained from 120 randomly-generated stroke sequences (17.2; SD 2.6; F[1,238]=3953, p< .001). This finding supports the proposal that the performance of normal subjects in copying the daisy is based on traversing a hierarchical representation like the one in Figure 2.
For a particular orientation of daisy, the probability that the part would be drawn (i.e., not neglected) in a particular frame was assumed to be a monotonically-increasing function of its horizontal position in that frame (Figure 3). The specific (exponential) form of this function is not critical as it influences only quantitative aspects of the results. Notice that the probability of drawing a part is near 1.0 on the right side of the frame, about 0.9 at the midline, and drops off sharply towards the left of the frame. The overall likelihood that a part is drawn was assumed to be a weighted average of its separate probabilities in the viewer-centered frame and in the object-centered frame---the effects of different relative weightings are explored below. All else being equal, the effect of neglect is generally stronger in the object-centered frame than in the viewer-centered frame because the former is defined more locally (i.e., parts typically fall outside the +/-1 frame defined by the horizontal extent of their parent).
A simple depth-first tree traversal algorithm was used to determine the neglect pattern. At every node, the probability that the corresponding part was drawn was calculated based on its viewer-centered and object-centered coordinates. We assumed that if a part was not drawn, then none of its subparts would be drawn. Thus, the probability of a part being drawn is the product of and the probability of its parent being drawn and its own local probability based on its relative positions in the viewer- and object-centered frames. The order of traversal among children of the same parent was irrelevant. The outcome of the tree traversal was that every part was assigned a probability of being drawn based on the orientation of the daisy and the particular weightings of the viewer- and object-centered frames.
Because the misoriented daisy allows for the decoupling of the viewer- and object-centered effects, unlike the upright daisy, Figure 4 illustrates the independent contribution of viewer-centered neglect and of object-centered neglect in a left-facing daisy. The numbers superimposed on the daisy indicate the probability of each feature being drawn, calculated according to the algorithm described above. It is important to recognize that the probability of a part being drawn is contingent on the probability of its parent being drawn---if the parent or containing objects is omitted, so is the child. The probabilities for the subparts such as the petals and leaves, therefore, reflect the conditional probability of parent and child both being drawn and are subsequently lower than the probability of the parent alone.
As is evident from this figure, when the viewer-centered influence is 1.0 with no object-centered influence (Figure 4a), information on the viewer-centered left has a fairly low probability of being drawn, with the probability of the daisy head being 0.75 and the surrounding left and right petals ranging from 0.45 to 0.63 respectively. The petal that occupies the leftmost position has a probability of 0.38. In contrast, when the viewer-centered effect is set to have no influence and neglect arises solely within the object-centered frame (Figure 4b), information to the right of the canonical midline of the daisy has a high probability of being drawn (approximately 0.94) whereas the petals and leaf on the left of the intrinsic axis have a very low probability of being drawn (approximately 0.24). The leaf on the canonical left stem has a probability of 0.06 because it is conditional on its parent stem being drawn and because it occupies the most extreme left position in the object-centered frame.
Figure 5 illustrates how a mixture of neglect in the viewer- and object-centered frames affect the probability of a feature being drawn. As is evident from the upright daisy, the object-centered effect has a somewhat stronger influence on performance---when it is set at 0.75, the probability of the petals and stem on the left being drawn is lower than when the viewer-centered effect is set at 0.75. In all cases, the midline structures (flowerhead, stem, and pot) have a high probability of being drawn. The slightly stronger influence of the object is also seen in the upside-down figure where, even when the two effects are equivalent at 0.5, the probability of drawing parts on the left, defined in the object-centered frame, is lower than the probability of drawing parts on the left of the viewer-centered frame (canonical right of daisy). The interaction of the two effects is best seen in the 90 deg misoriented daisies. In the left-facing daisy, the probability of drawing the petals on the canonical left (where the petals are also on the viewer-centered left) is much lower than in the right-facing daisy (where these same petals appear on the right of the viewer-centered frame). As either the severity of the viewer- or the object-centered neglect is increased, so the probability of drawing such a left-sided petal decreases. For example, on the left-facing daisy, when the viewer- and object-centered effects are equivalent, at 0.5, the probability of drawing the petal to the immediate left of the canonical midline is 0.36. As the object-centered effect is decreased to 0.25, the probability of that petal being drawn increases to 0.41, and as the object-centered effect is increased to 0.75, so the probability decreases to 0.30. The important point is that when a petal appears on the left of both the object- and the viewer-centered frame, it has a high probability of being neglected, relative to when it appears on the left in only one frame. This is in marked contrast to the data from the same petal (canonical left) on the right-facing daisy in which, irrespective of the relative contribution of object- and viewer-centered neglect, remains approximately constant with a probability of about 0.97. The crucial result from this analysis is that there are joint effects on neglect on the distribution of attention in both frames and that these are exacerbated as the severity of neglect---particularly in the object-frame---increases.
Figure 5:
The effects of various mixtures of neglect in the viewer-centered
frame and in the object-centered frame(s).
The probabilities in the figures can be interpreted as indicating the frequency with which individual parts would be drawn over a large sample of drawings. In order to account for the drawing performance of a single neglect patient, JM, and to generate daisies that would adequately characterize his performance, we adopted a discrete approach in which we converted the probabilities into absolute values. We then selected a threshold of 0.57 such that features with values below this threshold were not drawn (i.e., were neglected). Figure 6 contains the renderings of the daisies in four orientations when this threshold value is used. The relative weightings of viewer- and object-centered neglect were 0.6 and 0.4, respectively, as these best accounted for JM's drawings. As can be seen from these drawings, the data are reasonably well captured by this mixture of object- and viewer-centered neglect with the threshold as the cut-off value for performance. One apparent discrepancy is that JM's drawing of the left-facing daisy does not contain the upper-right petal (Figure 1b). As it turns out, he initially drew this petal and then erased it, removing a small part of the circle along with it. Another aspect of his data not yet explained is the right-facing daisy (Figure 1d), in which petals on the canonical object left are omitted but petals are also omitted from the left of the daisy head defined in viewer-centered coordinates. This pattern again reflects the joint effect of viewer- and object-centered effects. One interesting possibility is that after JM drew the daisy head (which he did first), because the circle in the center has no intrinsic axis, it has no obvious left-right asymmetry. This ambiguity lends itself to neglect in both frames: In an object-centered frame, a petal to the left is ignored and in a viewered-centered frame, petals to the left are also ignored. Although we cannot account for these data in our current implementation and the joint effect of the two frames in Figure 1d is beyond the interactions we have modeled here, the model may be extended to account for these data utilizing the same principles as those incorporated in the simulations thus far.
One aspect of neglect copying performance which is not easily explained by the current model and which is robust across patients is that some features are retained. Under the account we have proposed, these same features should probably be omitted. For example, patients almost never draw only the right half of the circle for the head of the flower, nor do they omit the lip of the pot (if the base is drawn), even if it occupies a position on the left of the spatial reference frame. Similarly, in clock drawing or copying, even if patients neglect to fill in the numbers on the left of the clock, they invariably draw the entire perimeter of the clock. A possible explanation for this retention of left-sided information derives from the nature of the object representation underlying the subject's performance. Several recent studies have shown that patients with neglect remain sensitive to Gestalt properties of the stimulus. Thus, if a feature on the left of the object's midline can be grouped together with a feature on the right to form a ``good'' figure, based on principles such as good continuation, symmetry or closure, the left-sided feature is less likely to be neglected (Ward, Goodrich, & Driver, 1994). The grouping of features according to Gestalt heuristics may be incorporated into the hierarchical representation adopted here and a rather direct extension of the current implementation can account for these seemingly contradictory results.
The approach adopted here has been to characterize systematically the behavior of a mechanism in which hierarchical object representations and multiple reference frames interact to co-determine performance of a system. The simulations are not intended to be a veridical instantiation of the neural mechanism underlying neglect nor to parallel directly the function of parietal lobe. The principles embodied in this work, however, are consistent with many views that argue that the parietal lobe integrates and transforms data from one set of coordinates to another (Colby, 1991, Karnath, 1994, Stein, 1992). How the brain might actually implement a hierarchical representation and how it might achieve the dynamic reassignment of the components to parts and wholes are difficult research issues (see Hinton, 1990, for a connectionist approach to these problems).
The example of the daisy was used in this research because it is standardly used in the clinical assessment of neglect and because much is known about the way neglect patients perform on this task. The principles governing the joint effects of neglect in more than one reference frame, as proposed here, however, are believed to apply more generally. Predictions from this model may be generated to account for the copying behavior of neglect patients when they are confronted with more complex hierarchical objects and visual scenes.
Caramazza, A., & Hillis, A. E. (1990).
Spatial representation of words in the brain implied by studies of a
unilateral neglect patient.
Nature, 346, 267-269.
Cohen, J. D., Romero, R. D., Servan-Schreiber, D., & Farah, M. J. (1994).
Mechanisms of spatial attention: The relation of macrostructure to
microstructure in parietal neglect.
Journal of Cognitive Neuroscience, 6(4), 377-387.
Colby, C. L. (1991).
The neuroanatomy and neurophysiology of attention.
Journal of Child Neurology, 6, S90-S118.
Driver, J., Baylis, G. C., Goodrich, S., & Rafal, R. D. (1994).
Axis-based neglect of visual shape.
Neuropsychologia, 32(11), 1353-1365.
Driver, J., & Halligan, P. W. (1991).
Can visual neglect operate in object-centered coordinates: An
affirmative study.
Cognitive Neuropsychology, 8, 475-496.
Farah, M. J., Brunn, J. L., Wong, A. B., Wallace, M., & Carpenter, P. (1990).
Frames of reference for the allocation of spatial attention:
Evidence from the neglect syndrome.
Neuropsychologia, 28, 335-347.
Gainotti, G., Messerli, P., & Tissot, R. (1972).
Qualitative analysis of unilateral spatial neglect in relation to
laterality of cerebral lesions.
Journal of Neurology, Neurosurgery, and Psychiatry, 35,
545-550.
Hinton, G. E. (1990).
Mapping part-whole hierarchies into connectionist networks.
Artificial Intelligence, 46(1), 47-76.
Humphreys, G. W., & Riddoch, M. J. (1994).
Attention to within-object and between-object spatial
representations: Multiple sites for visual selection.
Cognitive Neuropsychology, 11(2), 207-241.
Karnath, H. O. (1994).
Disturned coordinate transformation in the neural representation of
space as the crucial mechanism leading to neglect.
In P. W. Halligan, & J. C. Marshall (Eds.), Spatial neglect:
Position papers on theory and practice (pp. 147-150). Hove, UK: Lawrence
Erlbaum Associates.
Kinsbourne, M. (1993).
Orientational bias model of unilateral neglect: Evidence from
attentional gradients within hemispace.
In I. H. Robertson, & J. C. Marshall (Eds.), Unilateral
neglect: Clinical and experimental studies (pp. 63-86). Hove, UK:
Lawrence Erlbaum.
Kosslyn, S. M. (1987).
Seeing and imagining in the cerebral hemispheres: A computational
approach.
Psychological Review, 94(2), 148-175.
Marr, D. (1982).
Vision.
San Francisco, CA: W. H. Freeman.
Marr, D., & Nishihara, H. K. (1978).
Representation and recognition of the spatial organization of
three-dimensional shapes.
Proceedings of the Royal Society of London, Series B, 200, 269-294.
Marshall, J. C., & Halligan, P. W. (1993).
Visuo-spatial neglect: A new copying test to assess perceptual
parsing.
Journal of Neurology, 240, 37-40.
Mozer, M. C., & Behrmann, M. (1990).
On the interaction of selective attention and lexical knowledge: A
connectionist account of neglect dyslexia.
Journal of Cognitive Neuroscience, 2(2), 96-123.
Palmer, S. E. (1977).
Hierarchical structure in perceptual representation.
Cognitive Psychology, 9, 441-474.
Posner, M. I. (1988).
Structures and functions of selective attention.
In T. Boll, & B. Bryant (Eds.), Clinical neuropsychology and
brain function: Research, measurement and practice (pp. 173-202).
Washington, DC: American Psychological Association.
Stein, B. F. (1992).
The representation of egocentric space in the posterior parietal
cortex.
Behavioral and Brain Sciences, 16, 691-700.
Taylor, H. A., & Tversky, B. (1992).
Descriptions and depictions of environments.
Memory and Cognition, 20, 483-496.
Van Sommers, P. (1989).
A system for drawing and drawing-related neuropsychology.
Cognitive Neuropsychology, 6(2), 117-164.