kosslyn's island experiment

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

5 Lab 5. Mental Imagery: What Can You See in Your “Mind’s Eye?”

C OGLAB Exercise 38

Introduction

Mental imagery was a favorite topic of the classic introspectionists. In fact, one of the raging debates around the turn of the century was whether there was such a thing as “imageless thought.” It was widely accepted that most “thoughts” involved images (if you were thinking of an apple, you imagined an apple in your mind). However, some introspectionists held that all thought involved images. The debate was never really settled, as the rise of behaviorism brought an early end to the argument. Mental activity, according to the behaviorists, was not a relevant venue for study.

As the cognitive revolution took hold in the 1960s, the topic of mental imagery again became controversial. Ironically, though, the debate took a different form. Rather than debating whether thinking could occur without images, now the argument was whether or not there was such a thing as mental imagery! Some held that all thought was done verbally, and that there were no mental images.

On the surface, this might seem like an impossible debate. After all, how can we possibly know if there are images? We can’t very well open up somebody’s head and take a look! In the early 1970s, Roger Shepard devised a technique that, in a sense, did allow us to see what is going on with the images inside our heads.

Thought Question:

Try this example: Look at the letters below, and decide whether they are the normal letter, but rotated, or whether they are reversed (backwards) letters:

Do the Thought Question exercise above. You probably had little trouble realizing that the top row is comprised of normal letters, and the bottom row of reversed letters. But did you find that the middle three columns (the ones rotated either 90, 180, or 270 degrees) were somewhat more difficult? Most people do, and require the most time to make a decision when the letters are rotated 180 degrees.

Introspect for a moment: how did you perform the task above? Most people report that they somehow mentally rotate the letters until they are vertical in minds, and then make their judgment. The more the letter is rotated, the longer it will take to rotate it back to the proper position in your mind. Mental rotation exercises such as these have become a favorite way to investigate mental imagery.

The topic of mental imagery is tricky. We know that you don’t have perfect snapshots of objects in your mind. That’s just not the way memory works. How, then, are these mental images stored, and how do we operate upon them? According to Shepard (Shepard & Chipman, 1970; Shepard, 1978; Shepard & Metzler, 1971), images are not perfect copies of real objects, but second-order isomorphisms. This is a fancy term, but all it means is that the image you have is lawfully related to the real object. An isomorphism means that something has the same form (iso- same , morph- form ). A first-order isomorphism would be something like a photograph, where the two forms were highly similar (they were related on the “first-order”). A second-order form means that though the mental and physical objects are not identical, they are still similar—perhaps a cartoon rendition of a sunset, rather than an actual video of one. The way we mentally work on the internal object is related to the way we would physically work on the external object. If we have an inverted object in the real world, we physically rotate it. If we have an inverted mental object, we mentally rotate it.

Roger Shepard and his colleagues found evidence to support the concept of second-order isomorphisms in mental rotation. The time required to identify a stimulus was directly related to the stimuli’s “degree of rotation,” similar to the pattern shown in Figure 5.1. This finding has been replicated many times, and also extends into rotations in three dimensions (you may remember those 3-dimensional rotation tests from the standardized achievement tests often given in elementary schools).

Figure 5.1. A typical relationship between the time required to identify a stimulus and the degree of rotation.

This is not the only way to study mental imagery. For example, Kosslyn, Ball, and Reisser (1978) performed an experiment I refer to as the “Gilligan’s Island” experiment. They had subjects memorize the layout of a fictitious island that had various landmarks. Assume that the island looked like that shown in Figure 5.2.

Figure 5.2. A map (vaguely) resembling that memorized by the subjects in Kosslyn, Ball, and Reisser (Kosslyn et al., 1978)

Subjects studied the map for several minutes, memorizing the location of various objects. Then, Kosslyn et al. (1978) asked subjects to imagine they were “at” one location on the island–let’s say, for example, they were told to “go” to the boat in their minds. Once subjects said that they had mentally “located” the boat, they were given the name of a second location–let’s say, the “rock.” They were told to locate the second object as quickly as they could in their mental map of the island, and press a button when they were finished. The question that Kosslyn et al. (1978) were asking is similar to that asked by Shepard and Metzler. Does it take time to scan from one location to another in a mentally imagined environment? Kosslyn et al. (1978) then compared the scanning time for the mental image to the physical distance on the map. The relationship is shown in Figure 5.3.

Figure 5.3. Scanning times (milliseconds) as a function of the physical distance between the two points, adapted from Kosslyn et al. (1978)

As you can see, the time to scan the mental image is directly related to the physical distance between the two objects. Do you physically move from one location to the other in your mental map? That may be putting it too strongly; however, the concept of “second-order isomorphisms” allows us to say that while you might not physically move, you are doing something indirectly tied to the literal, physical representation 12 .

The experiment we are going to perform is similar to one first conducted by Shepard and Metzler (1971). You will be asked to judge whether two images are identical but rotated, or whether they are both rotated and reversed.

At this time, complete the experiment rotation of mental rotation in COGLAB . Instructions can be found in

Lab 38 of the COGLAB Website .

Questions for Lab 5

1. What is the independent variable in these studies? What is the dependent variable?

2. What things were controlled for in the present experiment? Why were they controlled? Can you think of other important variables that we didn’t control? How might we improve the experiment, controlling for those additional factors?

3 . Calculate the speed of rotation in degrees per second for both your individual and the class data. How does our figure compare with that of Shepard and Metzler? Why might there be some discrepancies?

4 . Why did Shepard & Metzler’s (1971) findings suggest that mental images were similar to real images?

5. In demonstrations like this, participants often report that they mentally rotate the objects in their minds in order to make their same/different judgments, so why did researches decide to take reaction time measurements also?

6. You are trying to get a new mattress through your front door. Currently, the mattress is laying flat on your front porch. You look at the mattress and mentally rotate the object in your mind to figure out how it might fit through the doorway. There are two ways the mattress could conceivably fit through the doorway: it could be slightly tilted to one side or it could be stood up on end. Using what you have learned in this demonstration, which of these two possibilities are you more likely to come up with first? Why?

7 . This experiment examined mental imagery. What about imagery for other senses? Give at least two examples of mental representations for other senses, specifically sound and taste/odor.

Data Sheet for Lab 5

Report Mean Percent Reaction time

Speed of Rotation: _______ degrees per second

Graphs for Lab 5

3-D Mental Rotation

Individual Data

Laboratory in Cognition Student Manual Copyright © by Charles Weaver, III. All Rights Reserved.

Share This Book

• Table of Contents
• New in this Archive
• Chronological
• Editorial Information
• About the SEP
• Editorial Board
• How to Cite the SEP
• Special Characters
• Support the SEP
• PDFs for SEP Friends
• Make a Donation
• SEPIA for Libraries
• Back to Entry
• Entry Contents
• Entry Bibliography
• Academic Tools
• Friends PDF Preview
• Author and Citation Info
• Back to Top

Supplement to Mental Imagery

The quasi-pictorial theory of imagery, and its problems.

The basic functional architecture of Kosslyn's theory is shown in figure 1. This depicts what might be happening when someone considers the question of whether a fox has pointed ears: by first constructing a mental image of a fox, and then examining the ears in the image. (This, for Kosslyn, is prototypical of the sorts of task for which imagery is normally used in cognition).

As the diagram shows, the data from which images may be constructed are taken to be stored in Long Term Memory in the form of deep representations , “propositional” descriptions of the visual scene, presumably not dissimilar to the sorts of description envisaged by Pylyshyn. [ 1 ] However, as Kosslyn sees it, these are not directly available to consciousness. They are analogous to the files in which data is saved by a computer graphics program, and on the basis of which actual, viewable pictures are constructed on the screen of the computer monitor. When we form a mental image a quasi-picture or surface representation is constructed, on the basis this stored information, on a functionally defined surface that Kosslyn calls the visual buffer . This buffer is also supposed to be a stage in perceptual visual information processing (similar in function to the 2½D sketch of Marr's (1982) theory of vision [ 2 ] ), when it would receive its input from the eyes instead of from the Long Term Memory store. Once the quasi-picture is established on the buffer it is available to consciousness as an image, and, furthermore, information that was merely implicit in the deep representation (such as the pointedness of the fox's ears, in the depicted example) can be extracted from it by a postulated mind's eye function . No doubt we are not supposed to think of this mind's eye as literally seeing the image, but it is needed to read and interpret the buffer's surface display . A principal function of images in cognition, as Kosslyn sees it, is to enable ready access to visuo-spatial features and relations of this sort, features or relations that might otherwise be very hard to infer from the “propositional” (mentalese) description.

Kosslyn & Shwartz (1977, 1978) developed a computer simulation model of key parts of Kosslyn's theory ( not including the mind's eye function ), and its instantiation of the surface representation , the model for the mental image itself, is revealing. Figure 2 was produced by selectively filling cells in a rectangular array or matrix, which models the visual buffer of the underlying theory. Note that certain cells contain different letters. Kosslyn plausibly assumes that images fade over time, and successive letters of the alphabet represent older, successively more faded portions of the image. The cells, then, are not simply filled or unfilled, they contain symbols representing qualitative features of the image at that point. It is what Tye (1991) calls a “symbol filled array”. The theory assumes that each cell might contain multiple symbols, representing such things as color, 3-D depth and the presence of edges (Tye, 1991; Kosslyn, 1994) (although none of this, beyond the degree of “fading”, was implemented in the simulation).

Figure 2 Simulated image ( surface representation ) of a car, as generated by the Kosslyn & Shwartz (1977) program.

The array representation is constructed from a “deep representation” description, stored in the computer's equivalent of long term memory (as in figure 1 – the program had just two “images” hand-coded in: a car, as shown in figure 2, and a chair, in similar vein), and can thus be readily manipulated in various ways. It can be redrawn at various different sizes, positions and orientations on the buffer, and with varying amounts of detail included. This allows Kosslyn to account for the rotation, scanning, and size/inspection time effects which particularly interest him (see section 4.3). He can also consistently explain the mnemonic properties of imagery via a version of Dual Coding theory (Kosslyn, Holyoak, & Huffman, 1976).

It is important to remember, however, that the surface representation is a quasi -picture or functional picture, rather than a picture in the fully literal sense (see Tye, 1991). The real model for the mental image is not what is displayed on the computer screen, or printed out by the program. These representations, after all, are not available for further processing (such as by some mind's eye function ) within the computer. They are just there for the benefit of human spectators. The real model for the mental image is the underlying array representation in the computer's memory, physically instantiated as electronic states in RAM chips. Image manipulations are carried out on this array, and merely mirrored on the screen or printout. But the array representation in the computer (and, its counterpart, if it exists, in the brain) is not in itself visible, like a real picture would be. Indeed, the representations of cells that appear adjacent in the screen display or printout need not even be in physically adjacent sections of RAM (though they might be). All that matters is that they be treated as if they were adjacent by the computational routines which form, transform and inspect the array (Kosslyn, 1980). [ 3 ]

Moreover, if, as suggested, each cell in the array holds several symbols, for various represented qualities, there is no need for even the symbols within a single cell to be represented in physically close memory elements. We might have multiple arrays, one for each sort of quality, so long as the accessing routines treat them as a single, superposed, array with multiple symbols in each cell. This is relevant, because in Kosslyn's more recent, neuropsychological version of the theory the visual buffer is taken to be composed of multiple arrays, instantiated in the multiple, specialized, retinotopically mapped areas of visual cortex (the primary visual cortex, V1, and other, adjacent visual areas) that are to be found in the brain's occipital cortex, at the back of the head (Kosslyn, 1994, 2005; Kosslyn, Ganis, & Thompson, 2001; Kosslyn, Thompson, & Ganis, 2006). [ 4 ]

An interesting variant of the two-dimensional array model of Kosslyn & Shwartz is proposed by Pinker (1988), who argues that a three-dimensional array will better incorporate the evidence that suggests that the spatial properties of imagery extend to the third dimension (Shepard & Metzler, 1971; Pinker & Kosslyn, 1978; Pinker, 1980; Pinker & Finke, 1980; Kerr, 1987). (Perhaps this might be called a quasi-sculptural theory.) However, this idea seems to have attracted little attention, perhaps because it appears to be inconsistent with Kosslyn's ideas about the neural retinotopic maps, which are necessarily two-dimensional.

The implementational details of the Kosslyn & Shwartz computer model, such as the fact that the points on the image are plotted with polar co-ordinates, do not matter very much. Nor, indeed, does the crudity of the images produced (as seen in figure 2). The point of the model is simply to provide a concrete illustration of the theory, and to show that, since it can be implemented, this part of the theory does not contain an undischarged homunculus (Dennett, 1978 ch. 7), that would render it non-explanatory. (Of course, the model does not show that there is no undischarged homunculus in unimplemented parts of the theory, such as the mind's eye function , which is where we would most likely expect him to be lurking.) Several different ways of computationally implementing (symbol filled) array theories of imagery have been proposed by AI researchers since the original Kosslyn & Shwartz work: some connectionist (e.g. Julstrom & Baron, 1985; Mel, 1986, 1990; Stucki & Pollack, 1992), and some not (e.g. Glasgow & Papadias, 1992; Glasgow, 1993). These all have interesting features, but none attempt the psychological range of Kosslyn's model. We must, of course, look to neuroscience for an account of how quasi-pictorial imagery might actually be implemented, and that is what Kosslyn has been doing over the last couple of decades (e.g., 1988, 1994, 2005; Kosslyn, Thompson, & Ganis, 2006).

There has been much dispute between experimentalists over whether the brain imaging evidence (mostly fMRI and PET scans) supports the contention that V1, the primary visual cortex (which is by far the largest retinotopically mapped area of the visual cortex, with, by some distance, the most detailed map) is preferentially activated during visual imagery. Certainly many experiments have found that V1 activity is elevated during imagery, but many others have found no sign of this, and instead find elevated activity in other brain areas, often ones that are not retinotopically mapped at all (see Kosslyn & Thompson, 2003, for review ). Kosslyn & Thompson (2003) suggest that these apparently conflicting findings might be reconciled if the activation level of V1 depends upon the sorts of task that the experimental subjects are being asked to do with their imagery: Tasks that do not require attention to the detailed geometric shape of imagined objects, they suggest, may not activate V1, but may still activate the other (low resolution) retinotopically mapped areas such as those responsible for processing color, or visual motion, or spatial layout information (see Mazard et al., 2004).

There is, however, a further empirical difficulty in the way of the identification of the retinotopically mapped areas with the visual buffer of Kosslyn's theory. It appears that people can experience vivid visual imagery even when at least some of the relevant parts of the retinotopically mapped areas in their brain have been damaged. There are many studies of people who are blind in all or part of their visual field precisely because the corresponding part of V1 (or sometimes other, nearby retinotopically mapped areas) has been destroyed, but who, nevertheless, seem either to have normal visual imagery, or, in some cases, to experience well-formed “hallucinations” (i.e., particularly vivid, spontaneous imagery) specifically in those parts of the visual field where they are thus blinded (Bridge et al. , 2012; Zago et al. , 2010; Dulin et al. , 2008; Ashwin & Tsaloumas, 2007; Kleiter et al. , 2007; Ramachandran & Hirstein, 1997; Roland & Gulyàs, 1994; Chatterjee & Southwood, 1995; Goldenberg et al ., 1995; Grüsser & Landis, 1991 pp. 156-157; Kölmel, 1985; Lance, 1976; see also Servos & Goodale, 1995). This strongly implies that, even if imagery is sometimes accompanied by activation of the V1 map (and/or other occipital retinotopically mapped visual areas), it cannot be dependent upon (let alone identical with ) such activation.

But even if it were a fact that retinotopically mapped areas of the brain are always active during visual imagery (which , as we have just seen, it is not), is that strong evidence in favor the quasi-pictorial theory? Clearly Kosslyn himself thinks so, but Pylyshyn still demurs. The issue, he points out (Pylyshyn, 2002b, 2003b; see also: Abell & Currie, 1999; Thomas, 1999b; Dennett, 2002), is not whether certain 2-D patterns of activation (corresponding to potential patterns of retinal stimulation) exist in the brain, but whether such patterns ( qua patterns) play a causal role in generating the behaviors that we see in experiments on imagery (and, we might add, although Pylyshyn does not, in generating quasi-visual experiences). Elevated activity in the retinotopic maps (if and when when it does occur) might be no more than a side-effect of the brain processes that are truly responsible for imagery. Pylyshyn argues that, in fact, the amount of information that can be represented at any one moment in the retinotopic maps is not sufficient to support most of the experimental imagery effects that quasi-pictorial theory purports to explain (Pylyshyn, 2002b; cf. Ingle, 2002). Indeed, during visual perception itself the information contained in the retinotopic maps at any one moment is not sufficient to explain our experience of the visual scene, or, indeed, our capacity to visually guide our behavior (O'Regan, 1992; O'Regan & Noë, 2001; Noë, 2004). Seeing is not like taking a snapshot; it is a process that is extended in time. It is true that it is possible to treat the primary visual cortex as if it were a sort of photographic plate, with, at any one moment, a (distorted [ 5 ] and low resolution) snapshot of the retinal image latent within it (Tootell et al. , 1982). However, we experience neither vision nor imagery as though they were a succession of such snapshots.

Quasi-pictorial theory also faces further empirical challenges. For one thing, all the major experimental effects that supposedly reveal the spatial and non-verbal properties of visual imagery (such as mental rotation, scanning, size/inspection time effects, and selective interference), have now been demonstrated in totally congenitally blind subjects (Marmor & Zaback, 1976; Carpenter & Eisenberg, 1978; Kerr, 1983; Hampson & Duffy, 1984; Vecchi, 1998; Barolo et al ., 1990; Aleman et al. , 2001). [ 6 ] As the stimuli in most of these experiments were presented haptically (i.e., via the sense of touch), the blind subjects seem likely to be using haptic (touch based) imagery to do the experimental tasks (see Klatzky et al. , 1991; Prather & Sathian, 2002). The trouble is that there seems little prospect of a haptic equivalent of the quasi-pictorial theory of visual imagery (Thomas, 1999b). A “passive-touch buffer” (mapping the skin much as the visual buffer is supposed to map the retina) may be conceivable, but it would not be remotely adequate to the tasks in question. Haptically based knowledge of objects and spatial relations is clearly mediated not just by touch sensations, but by active, exploratory movements, involving a complex coordination between tactile sense, proprioception, and motor control (Davidson, 1972; Lederman & Klatzky, 1990). Any theory of haptic mental imagery would surely need to incorporate this fact, and although it is conceivable that visual imagery might work in one (quasi-pictorial) way and haptic imagery in some quite different (presumably enactive) way, if that were so the formal similarity between the experimental results from the two populations (congenitally blind and sighted) would be a very strange coincidence.

Another significant empirical challenge comes from the fact that experiments have shown that it is much more difficult for people to reinterpret or re-construe the structures or meanings of their mental images than it is for them to reinterpret equivalent pictures. Images seem to incorporate their interpretation in a way that pictures do not: As Wittgenstein remarked, “the concept ‘I am now seeing it as ….’ is akin to ‘I am now having this image’.” (Wittgenstein, 1953 p. 213 – ellipsis and emphasis in original). Several experiments have demonstrated this, but perhaps the most striking is that of Chambers & Reisberg (1985; see also Chambers & Reisberg, 1992) in which subjects attempted to reinterpret their mental images of bistable ambiguous figures (i.e., figures with two more or less equally plausible interpretations) such as the Necker cube and the duck-rabbit (figure 3).

Chambers & Reisberg (1985) first introduced their subjects to the concept of bistable ambiguous figures by showing them some examples, then showed them one of the figures that they had not previously seen, but only for 5 seconds, too short a time for them to see more than one of the possible interpretations. They were then asked to form a mental image of the figure they had just seen, and to try to find a second interpretation in their image. Despite having plenty of time, and being given hints and encouragement by the experimenters, in no trial (out of 55 in all) did any of the subjects manage to see or even guess the alternative interpretation. Even more strikingly, when the subjects were then asked to draw the figures they had seen, on the basis of the image they had formed, in the vast majority of cases they were soon able to see the alternative interpretation in their own drawing.

It should be admitted that in other experiments it has been shown that the reinterpretation or reconstrual of images is possible under some circumstances (Finke et al ., 1989; Reisberg & Chambers, 1991; Peterson et al ., 1992; Brandimonte & Gerbino, 1993; Cornoldi et al. , 1996; Mast & Kosslyn, 2002a; Thompson et al. , 2008). Nevertheless, this does nothing to undermine the integrity of the original results, and others that point in the same direction (Reed, 1974; Palmer, 1977; Reisberg et al. , 1989; Slezak, 1991, 1992, 1995). People clearly experience significant difficulties in reconstruing their mental images under conditions where they have very little trouble with the equivalent picture. Images differ from pictures because they seem to carry their interpretation within them in a way that pictures (even quasi-pictures) do not.

The quasi-pictorial theorist does have a comeback, however. For largely independent reasons, Fodor (1975) suggested that if images have a function in cognition, it must be as what he called images under descriptions . To a first approximation, the idea is that every mental picture comes tagged with a sort of mentalese caption, telling us what it represents. Tye (1991), who explicitly supports a quasi-pictorial account of imagery, takes a very similar view. Thus, a subject in the Chambers & Reisberg (1985) experiment who is briefly shown the duck-rabbit figure might initially form a quasi-picture of the figure tagged with the caption DUCK, and then be unable to avoid interpreting it except in the light of this description. (Presumably, if someone had the chance to view the figure for more than a brief moment, as in most non-experimental situations, they might have time to form two picture-description pairs, and could thus later imagine it either way.) Although this seems rather ad hoc (why should we not be able to freely dissociate and re-associate images and their descriptions?), Chambers & Reisberg (1992; Chambers, 1993; Reisberg, 1994) seem to be inclined to interpret their results in broadly this fashion (see also Rollins, 1994).

Fodor originally proposed that images always function in cognition under a description in order to account for the intentionality of mental pictures. Following Wittgenstein (1953 §139f; and see also Goodman, 1968), Fodor argued that pictures as such do not carry their interpretations within themselves (rather, a person has to interpret them to see what they are pictures of), and that, therefore, mental pictures as such cannot be intrinsically meaningful (bearers of original intentionality ). Thus, Fodor concluded, mental images cannot, in and of themselves, function as mental representations. However, he was impressed by the psychological evidence suggesting that images do function as representations, and proposed that they should therefore be understood not as bare pictures, but as pictures under (mentalese) descriptions, inheriting their meaningfulness or intentionality from the mentalese caption. This move solves an important problem, and Kosslyn (although he has taken little interest in the matter of image intentionality) seems to be prepared to countenance it (Kosslyn 1980 p. 449ff, 1994 p.6). However, it also means that quasi-pictorial theory becomes beholden to the popular but controversial language of thought hypothesis , and inherits all its problems and implausibilities (Hauser, 1995). Not the least of these is that, despite years of strenuous efforts to reach one, there is no consensus as to how mentalese representations themselves could come to have original intentionality, and there is some reason to think that they could not (Horst, 1996, 1999; Cummins, 1997).

Copyright © 2014 by Nigel J.T. Thomas < njtthomas @ yahoo . com >

Support SEP

Mirror sites.

View this site from another server:

• Info about mirror sites

The Stanford Encyclopedia of Philosophy is copyright © 2016 by The Metaphysics Research Lab , Center for the Study of Language and Information (CSLI), Stanford University

Library of Congress Catalog Data: ISSN 1095-5054