Creative Practice and Experimental Method in Electronic Literature and Human Experimental Psychology

Abstract:

This article discusses issues arising from the relationship between practitioners in Electronic Literature and researchers in the field of Human Experimental Psychology, including the possible emergence of new communities that cross over this boundary. The introduction (1) considers the possible drivers of this process, including technology, interdisciplinarity and research funding policy, after first explaining the source of the article in an interdisciplinary project, Poetry Beyond Text: Vision, Text and Cognition (2009-11). This project involved literary critics, psychologists and creative artists and studied works that combine (poetic) text with images, including digital poetry, concrete poetry, artists’ books, visual poetry and poetry-photographic works. In section 2 we discuss the concept of the “experimental” in aesthetic and scientific contexts, identifying the relatively universal model of the subject constructed through experimental procedure in Psychology and contrasting it with the radical idea of the subject implied by avant-garde aesthetic practice. We then discuss several examples of parallels between the methods of Electronic Literature and Experimental Psychology. Section 3 compares the flash works of Young-Hae Chang Heavy Industries and the psychological experimental technique of Rapid Serial Visual Presentation. Section 4 compares the visual poetics of digital poetry in the tradition of concrete / visual poetry (including John Cayley’s Translation and Jim Andrews’s Stir Fry Texts) with the manipulations of font and layout in psycholinguistic method. Section 5 compares John Cayley’s Lens, created in the virtual reality CAVE at Brown University, with the Mental Rotation test used in Experimental Psychology, referring to Cayley’s concept of the “phenomenology of the object”. Section 6 discusses in more detail a digital literary-visual artwork created for a single-screen 3D simulator, and commissioned as part of Poetry Beyond Text. Tower, by Simon Biggs and Mark Shovman, explores perceptual and cognitive processes in reading and is described as an “immersive 3D textual environment combining visualisation, speech recognition and predictive text algorithms”. It is here used as a case study for the interaction of digital poets / artists with psychologists and psychological findings, drawing on material from interviews and discussions with the artist and programmer involved, in particular Biggs’s interest in third-order cybernetics. The discussion deals with the construction of value around the concept of “interactivity” and the construction of the reader / viewer / subject. The conclusion (7) considers possible models for the relationship between creative practice in digital media and Human Experimental Psychology, addressing the conflict or convergence of ideological and epistemological values and assumptions.

_____________________________________________

1. Introduction 

In this article we consider issues of theory and practice arising from the relationship between two communities: practitioners in electronic literature and researchers in the field of human experimental psychology. These two disciplines are separated by seemingly different aims (aesthetic as against scientific), values (creative and interrogative as against descriptive and analytical) and sub-cultures (a strong creative community with a rich inheritance but a still emergent status as against an academic-scientific community with strongly established protocols, driven by competing theoretical views but also influenced by a rapidly-changing socio-economic environment). Yet convergences in respect of the questions they address and the methods they use suggest the potential emergence of new communities embracing both fields.

Our analysis spring from an interdisciplinary project, Poetry Beyond Text: Vision, Text and Cognition (2009-11), funded by the UK Arts and Humanities Research Council as part of their Beyond Text Programme, 2009-2012, and involving researchers from English Literature, Comparative Literature, Psychology, and Fine Art at the University of Dundee and the University of Kent.1. We studied critical, cognitive and creative responses to works combining text and image, including concrete poetry, artists’ books, visual poetry and poetry-photographic works as well as, crucially for the present argument, digital poetry. The aims of the project included: (1) To integrate the methods of literary criticism, experimental psychology, and practise-based research in fine art and poetry, so as to open up new possibilities for interdisciplinary research in the arts and humanities; (2) To generate a dynamic dialogue between empirical, theoretical-historical and creative modes of understanding, and investigate methodological and theoretical issues arising from this dialogue. One of the inspirations for Poetry Beyond Text, and one of its meta-methodological facets, has been the strong connections which have emerged between psychological experiment and creative practice – connections marked by some striking similarities in technology, procedure and focus of interest, combined with disparities in the underlying epistemological assumptions. Are we witnessing a convergence between these disparate fields, driven by technological change, intellectual agendas and the policy of governments and other funders of research? Or are these connections merely local and transitory moments of cross-fertilisation, when persistent radical differences of intent are temporarily diminished in the appeal of new and different models? 

Our conclusions will necessarily be provisional, for at least two reasons. First, both these areas of practice are changing rapidly, in part because of the very technological and other factors to which we shall refer in this article. The inter-relationships in which we are interested are also taking place across a much wider field, between creative practice generally in new and emergent media, and diverse forms of scientific and technical research, including neuroscience, human-computer interaction and artificial intelligence studies. However, electronic literature and human experimental psychology have a particular set of connections, to be discussed here, because they share specific interests in certain aspects of perception, cognition and motor skills.

We are not referring here to the much-discussed phenomenon of “digital convergence” (the coding of multiple media in digital format). Nor do we mean Henry Jenkins’s concept of “convergence culture”, “the flow of content across multiple media platforms, the cooperation between multiple media industries, and the migratory behaviour of media audiences”, empowered by a “participatory culture” (Jenkins). No doubt there are connections to both of these processes, but what we are referring to is a putative convergence in certain interests and methods between creators of Electronic Literature and researchers in academic Human Experimental Psychology (hereafter referred to as HEP). These interests include in particular the cognitive and emotive aspects of human-computer interaction and the relation between different perceptual and cognitive processes (such as visual and motor processes). The methods arise from the developing potential of technologies in imaging, simulation and two-way control between human and digital systems. Electronic Literature has connections particularly with those fields of psychological research concerned with, or using, text, notably psycholinguistics.

The drivers of this putative process of convergence or exchange include developments in technology, the growth of interdisciplinarity, and trends in research funding policy. As regards the first of these, and without espousing technological determinism, it is clear that the evolving technology of human-computer interaction in particular offers continually expanding possibilities for staging processes in which the human senses of sight, touch, hearing and proprioception are engaged in more-or-less two-way processes with progressively more intelligent or sensitive machines. Such processes can be used to study humans for a range of purposes, as in the extensive field of medical imaging, or the more “interactive” forms of psychological testing. They can also be used to offer experiences to humans (in virtual reality, games and digital art forms), some of which we might be inclined to term aesthetic experiences. There is clearly considerable cross-over, in which technology designed with one of these in mind offers new potential for the other (as when the results of advanced medical imaging are used in art works, or when game-like processes are used to study human behaviour). The other two drivers mentioned, interdisciplinarity and research funding policy, are closely interconnected: interdisciplinarity has been a favoured objective and criteria for the funding of research at both national level (for example the British Arts and Humanities Research Council) and supranational level (for example HERA, the European funding mechanism). Combined with the rise of the idea of creative practice as a form of research (itself driven by government funding policies), this has led to a promotion of practices that bring together creative practice and/or the arts and humanities with the sciences and social sciences, including psychology.

Following some comments on the idea of the “experimental” in scientific and aesthetic contexts, we will begin by looking at examples of parallels between the methods of electronic literature and experimental psychology: between the Flash works of Young-Hae Chang Heavy Industries and HEP’s experimental technique of Rapid Serial Visual Presentation; between the visual poetics of digital poetry in the tradition of Concrete / Visual poetry (Jim Andrews and Brian Kim Stefans) and the manipulations of font and layout in perceptual psychological experiments; and between John Cayley’s work for the four-wall VR Cave at Brown University, entitled Lens (2006) and the use of letter forms to measure spatial ability. We will then explore in more detail Tower (2011), by Simon Biggs and Mark Shovman, a work commissioned for Poetry Beyond Text. This work will be used as a case study for the interaction of digital poets/artists with psychologists and psychological findings. The discussion addresses issues such as the idea of the “experimental” (in its aesthetic and empirical senses), the construction of value and subjectivity around the concept of “interactivity”, and the conflict or convergence of ideological and epistemological values and assumptions.

2. The “Experimental” as Aesthetic and Scientific Concept

How valid is it to link creative practice and HEP in terms of the concept of “experiment”? Clearly, such a link should not be allowed to simplify the matter with respect to either field. The nature and status of the “experimental” is a topic of substantial and often complex debate in both aesthetics and the philosophy of science. In the broadest terms, an experiment suggests the staging of some process in order to “see what happens”. This could be applied to both fields, but with major differences. Psychological research typically has a pre-arranged dimension of evidence: both what is to be measured and the scale of that measurement are decided in advance. “Experimental” creative practice also involves designing a process or context for a human experience, but “what might happen” is left more open—and is not usually measured in any explicit way. This raises the question of what the feedback loop is in such creative practice – how does the “result” of experimentation feed back into future work? Katherine Hayles suggests that “[l]iterature … activates a recursive feedback loop between knowledge realized in the body through gesture, ritual, performance, posture and enactment, and knowledge realized in the neocortex as conscious and explicit articulations” (132). She sees literature as transforming implicit (bodily) knowledge into conscious knowledge, but also calling into question conscious knowledge. In this instance, the feedback is not a flow from the participant and apparatus which is then managed by the experimenter (the scientist or artist), but an effect which can presumably be experienced both by participant / reader and artist. Hayles’s comment is an instance of the way in which critical and theoretical writing on new media practice sometimes applies the language of scientific experiment to the analysis of creative practice, suggesting a convergence in discourse. Other examples are found in the work of new media theorist Mark Hansen, who writes about “cognitive activity” (3) and “the flow of data” (2), and suggests that:

by placing the embodied viewer-participant into a circuit with information, the installations and environments [created by contemporary media artists] … function as laboratories for the conversion of information into corporeally apprehensible images
(Hansen 11; our emphasis)

The experimental situation in HEP inevitably constructs a certain model of the subject through elements of constraint and manipulation. The aim is to isolate an individual variable and structure the experimental situation in order to maximize the visibility of this variable. This requires a high level of pre-assumption in terms of the model of behaviour. The need for such assumptions is reflected in the methodological demand for a hypothesis which the experiment tests. In general terms, the subject as postulated by HEP tends towards a universal brain / body. This is not to deny that HEP recognizes the existence of social and cultural factors in behaviour: it does this explicitly, and to some extent aims either to control them or to test for them. Nevertheless, in order both to achieve statistically significant results and to meet the scientific imperative towards general laws, the drive of areas of the discipline such as psycholinguistics is towards the general and universal (the case is different some sub-disciplines, such as social and personality psychology).

The experimental method relies on a series of methodological steps to derive new knowledge. These steps are grounded in the hypothetico-deductive method and based on arguments from formal logic. In what follows we give a very brief overview of this approach; a fuller description of the method can be found in an introductory HEP textbook such as that of MacLin and Solso. The typical starting point for an experiment is a critical reflection about a theoretical framework that consists of a series of consistent and related statements about an empirical domain, such as the origins and consequences of certain mental states. Such a theory is interrogated to deduce from it a prediction of the form “If A then B” (for example, “if a person is frustrated then this person will become aggressive”). This prediction of the theory is also called a hypothesis, and it states that, whenever the experimenter succeeds in creating A, then B should be observed. A and B can be directly observable behaviours or indirectly observable mental states. In either case there must be agreed procedures as to what constitutes A and B. These procedures describe the quantification of A and B and lead to statistical hypotheses about the quantitative relationship between A and B. To test such a specific hypothesis, the experimenter creates circumstances that constitute A and aims to measure B. If A leads to B the theory is supported. If A does not lead to B the theory is falsified on logical grounds, but it can be defended, for example by postulating that B occurs only after a delay that exceeds the measurement interval in the experiment. If B is present without A, logical reasoning prevents any conclusions about the theory. 

The human subject of aesthetic “experiments”—in the sense of “experimental” literary forms and practice, often associated with the avant-garde—is a potentially revolutionary or radicalisable political and social subject. Many experimental avant-garde poets, including the Dadaists Hugo Ball, Raoul Hausmann, and Tristan Tzara, the Surrealist André Breton, and post-war concrete poets, such as those belonging to the Wiener Gruppe and the Noigandres group, as well as Franz Mon, Reinhard Döhl, and Max Bense, were operating on the assumption that poetic, linguistic and disciplinary transgressions enacted in their poetries would generate psychological and ideological changes in their readers. By breaking the linguistic contract and unmasking its arbitrary and convention-based foundation, the encounter with the avant-garde text is to create rupture, to “shock” readers, to make them question their habitual perception strategies and assumptions about literature, language and, crucially, more wide-ranging socio-political conventions by implication. This belief in the psycho-political transformative power of experimental literatures has been developed further in the realm of post-structuralist theory, most notably by thinkers such as Julia Kristeva, Roland Barthes, Jean Baudrillard, and Michel Foucault. What is at stake in experimental literature, according to them, is not just an exploration of new possibilities for poetry and thought that lie beyond the cognitive boundaries imposed by “ordinary” language, but also a potentially socio-politically revolutionary insight into the constructed nature of our value systems and social conventions.

 

3. Young-Hae Chang Heavy Industries / Rapid Serial Visual Presentation 

A definition of reading from the perspective of HEP would be the process of extracting meaning from visually presented language. Since text is usually spatially distributed we must scan the text with our eyes. Eye movements during reading are short jumps, usually from one word to the next, and these oculomotor movements are interrupted by periods of rest during which we can extract information about the word we look at, as well as the word to the right. Lexical information is processed up to the semantic level and integrated with previously acquired information to form a coherent representation of the text. This process becomes more efficient as we learn to read, with skilled readers processing around 300 words per minute if they can accurately direct their eyes near the centre of each word, avoid frequent regressions to previously inspected text, and occasionally skip over words that were entirely processed while looking at the preceding word. From this description it is clear that normal reading involves not just lexical and semantic processing but also oculomotor control. In fact, the need to plan and perform repeated targeted eye movements is a major limiting factor for efficient reading. Within the discipline of HEP (and, more specifically, Psycholinguistics), reading has been studied without the limitations of this oculomotor component by presenting text word by word (or in short phrases) sequentially at the same place on a computer screen. This method has become known as rapid serial visual presentation or RSVP (Cocklin et al.; Potter et al.). Interest in RSVP was sparked by two ideas. First, it was thought that removing the need for oculomotor processing increased reading speed substantially. Some authors found that readers could comprehend text even at presentation speeds exceeding 1,000 words per minute or seventeen words per second, whereas just three or four words per second would be typical of normal reading (Juola et al.). Thus, using RSVP as a tool to study reading might lead to very different time line estimates for the various cognitive processes involved, when compared to more traditional methods. A second reason for the interest in RSVP is the possibility of presenting information on small displays, such as mobile phone screens. This work is driven by the need to develop user-friendly yet flexible communication tools (see for example Oequist and Goldstein).

The digital literary works of the duo of artists known as Young-Hae Chang Heavy Industries (Marc Voge and Young-hae Chang) use, as a formal literary technique, an effect comparable to the experimental technique of RSVP. Their web-based works are all programmed in Adobe Flash and presented as a series of words or short phrases which appear, rapidly and sequentially, in Monaco font, normally synchronized to music (often instrumental jazz). The parallel with experimental RSVP is far from exact. YHCHI’s works use a wide range of effects, and there is considerable variation in presentation: sometimes a phrase will build up in succession on a single screen before being displaced by the next (for example the opening of Lotus Blossom); sometimes a word will pulse, appearing and disappearing several times (for example in Beckett’s Bounce); sometimes the words on screen will wobble or appear to recede into the distance (for example in Dakota). Furthermore, the music clearly contributes powerfully to the overall effect of a dynamic, intense and engaging experience. Nevertheless, there are basic aspects of the experience which are shared by these works and RSVP as used experimentally: primarily the requirement to read text sequentially on screen in a rapid sequence of words or phrases, and the lack of control on the part of the viewer—the Flash programming of YHCHI’s work means that the reader / viewer has no control over the speed and cannot pause or go back (except to restart the work from its beginning). The unfamiliarity and processing difficulty created by this mode of presentation generates an experience for the viewer which is in some ways unpleasant or oppressive, though in other ways compelling and exciting. The works of YHCHI, to a greater extent than many works of digital poetry or literature, do ultimately foreground content, in that much of their interest arises from “traditional” literary effects and devices, such as narrative drive, irony, humour, poetic language and “voice” or narrative address. The sense of disruption and discomfort arising from the presentation mode is often used thematically, so that form reflects and supports content. In Dakota, for example, the speeding, jerky and sometimes shaking text mimics both the physical process and the alienated, desperate mood of a teenage road trip, while in Nippon the ironic narrative surrounding a Japanese woman entertaining businessmen in a bar has embarrassment, awkwardness and shame as significant themes, with the variations in speed suggesting these responses as well as the ebb and flow of talk and feeling in a long, late-night session.

Critics tend to assume that the speed at which the text passes is sometimes too rapid for effective reading. Jessica Pressman, for example, comments that “[Dakota’s] text flashes so fast that it is often impossible to read” (Pressman 305), and that

As is particularly and painfully obvious to Dakota’s dry-eyed and unblinking reader, speed is used as a technical tool to enhance the work’s difficulty. The use of difficulty as an aesthetic strategy bonds Dakota to modernism and the kind of reading practices its literature fostered. (Pressman 302)

Katherine Hayles observes that “the work proceeds at speeds rarely coinciding with a comfortable reading rate, either lingering longer than the reading requires or flashing by so quickly one must strain to catch all the words” (Hayles 125).

However, as noted above, reading research has in general found that RSVP presentation facilitates faster reading. YHCHI’s works are never, in fact, impossible to read as such; rather they challenge our sense of control and our wish to reflect or perform higher-level processing. Hayles herself makes the interesting link between YHCHI’s work and the “Speeder Reader” machine of the Research in Experimental Documents (RED) group at Xerox PARC in the course of an argument linking flashing text to modernity (Hayles 125–6). Here is how that group described itself: 

Research in Experimental Documents (RED) is the moniker for a small, interdisciplinary group at Xerox PARC. We are eight researchers involved in the creation of new genres based on emerging media and technologies. (Balsamo et al.)

The work of RED would seem to occupy an (emergent) intermediate space between creative practice, scientific research and technology marketing. As some of the group members comment: 

One major point of the entire XRF exhibit is to associate a sense of excitement, fun, and personal control with the idea of reading. So, for Speeder-Reader, we built a speed-racing interface onto speed-reading software…. [RSVP] has been used in several products as a speed-reading technology, and is sometimes used as a research tool by neurologists and perceptual psychologists. (Back et al. 623) 

Here we see “technologies of reading”, based on the RSVP technique, being developed as potential aids to human reading, as tools of psychological research, and as a form of entertainment aimed at promoting reading. The exhibition mentioned here (XFR: Experiments in the Future of Reading) took place at the Tech Museum of Innovation in San Jose in 2000, and included both machines which read and machines which humans use in order to read. Like the work of YHCHI, this brings us up against “an age-old dichotomy in our understanding of the automat (the machine)” which new-media theorist Lisbeth Klastrup (paraphrasing Erkki Huhtamo) describes as follows: 

we have perceived it either as a system, which allows control of and intervention into human activity (an automated system, a dehumanising machine), or as a system function, which relieves us of the triviality of repeated actions (an activity initiated by man, an extension of him). Hence, the process of interacting with the machine can be understood as an either liberating or restrictive activity, giving us, the humans, more control of the world or taking it away from us. Thus, interaction as activity either allows us control – or allows the control of us. (3)

YHCHI’s work is clearly not a psychological experiment in any strict scientific sense, since the reader’s responses and processes are not measured. But it has something of the feel of taking part in an experiment, because it seems to be testing (in the colloquial sense of that word) our perceptual and cognitive abilities. We experience a tension between the element of control involved in imaginative response, and a sense of being controlled by the work’s dynamic progression. Watching the work makes us self-conscious about our habitual reading and processing strategies in the way that taking part in a psychological experiment (such as eye-tracking) is liable to do. One accepted element of literature, or the literary, is the foregrounding of form and language: literature makes us aware of processes of interpretation, and the qualities of language itself. In that sense, YHCHI’s work can be located in the tradition of the “literary”. But, in addition to this, it makes us very aware of the more primary levels of processing necessarily prior to high-level interpretation: the movement of our eyes, the processing of visual information and the parsing of sentences. The dichotomy which Huhtamo perceives in terms of control has parallels to the venerable literary-critical debate about meaning: does meaning lie in the author’s mind, the text, or the reader’s construction?2 But with YHCHI, as with other electronic literature, the locus of the tension shifts from meaning to process: does the author / text / programme control our reading, or can we use our sense-processing and cognitive abilities to master that process? While it has always been the case that time-based art forms, such as cinema, dictate the speed of initial reception, most films have been edited so as to promote immersion in content and / or form (rather than the materiality of the film and the basic-level processing of visual and aural signals). The obvious exceptions are those avant-garde films which are precursors of the work of YHCHI.3 Pressman’s link to the aesthetic strategies of modernism is not entirely convincing. The difficulty of reading Pound’s Cantos (a work which is referenced in Dakota, as Pressman points out), is substantially a matter of content: multiple and various allusions and quotations; the use of several different languages; density and obliquity of thought; diversity of style. It is also a matter of form, both at the linguistic level (unconventional or fractured syntax) and at the literary level (a programmatic generic complexity). In the case of YHCHI it is really only the mode of physical presentation which is “difficult”: one might want to call this a formal difficulty, though it is form in a more physical sense. Written out on paper, the works of YHCHI would lose much of their effect; they would also be relatively straightforward in linguistic and literary terms, although still full of irony, humour and a certain degree of allusiveness. Furthermore Pound’s form invites one to read slowly with many pauses and rereadings, whereas YHCHI forces one to read quickly, as a result of perceptual rather than conceptual “difficulty”. Digital poetry can present many different obstacles and demands in relation to the physical and perceptual process of reading, and it is debatable how much purchase the concept of modernism gives us here.

4. Visual Poetics and Experimental Manipulations of Font and Layout

Reading researchers in HEP have manipulated both perceptual and cognitive-semantic aspects of texts to gain insights into the elementary processes involved in reading. Such manipulations are in some instances akin to those used in electronic literature for formal and aesthetic purposes – a resemblance which foregrounds the way in which such literature may work by challenging our habits of perception and cognition. In the context of HEP reading is conceptualised as a complex orchestration of perceptual, cognitive and motor processes in the service of extracting meaning from print. Perceptual processes include the encoding and identification of letters, while cognitive processes include grapheme-to-phoneme mapping for letters and letter combinations, semantic comprehension of words and syntactic ambiguity resolution across entire sentences. Finally, motor processes involve the planning and execution of targeted eye movements from one word to the next in order to integrate the spatially distributed information.

At the perceptual level, manipulations of line length, font, and contrast polarity (white characters on black background or vice versa) have been used in experiments aimed at improving simple legibility. For example, we know from classical work by Tinker that readers benefit from medium-long lines and that high positive contrast polarity (black on white) is more reader-friendly than high negative contrast polarity (white on black) or low contrast (grey text). Contrast manipulations have been used to study the interplay between overt attention shifts (eye movements) and so-called covert attention shifts (which occur independent of eye movements and refer to our ability to process information from peripheral vision). Differences in contrast or contrast polarity are common in works of electronic literature (especially the more “poetic” works): for example John Cayley’s Translation has (at certain points) off-white text on black background on the right of the screen, and grey text on white blocks of background to the left; I, You, We, by Dan Waber and Jason Pimble, uses a range of intensities and colourings of font; Michael Joyce’s Twelve Blue uses pale blue font on dark blue background; Jörg Piringer’s soundpoems uses black and pale grey letters on a white background.

 

Fig. 1. Screenshot of Translation, by John Cayley.

An HEP use of contrast effects is found in a study by Reingold and Rayner, who tested whether our covert attention is allocated serially to one word at a time or in parallel across multiple words within a line of text. They presented readers with normal black text on white background that included a single low contrast (faint) word. Once this word was fixated the authors measured the time it took readers to process the next word, also referred to as parafoveal word. If covert attention is deployed serially then the perceptual difficulty of a faint word should not affect processing of the parafoveal (normal) word. This is so because we would only attend to it once the fixated word has been successfully encoded, regardless of whether this is a fast or slow process. However, if we attend to multiple words at the same time, then two alternative predictions emerge: either the perceptual difficulty induced by the low contrast word should withdraw attentional resources from the simultaneous processing of other words, thus prolonging the dwell time of the eyes on the subsequent parafoveal word; or the longer dwell time on the faint word might enable more extensive parallel pre-processing of the parafoveal word, thus reducing the time it would subsequently take to read it. Consistent with the serial model of attention allocation, the authors found that there was indeed a strong effect of perceptual degradation on the dwell time on the faint word, but no penalty for processing the parafoveal word once it was fixated. Part of the point of this experiment was to support a model of the reading process (“the EZ Reader Model”) which distinguishes between “two stages of lexical processing: an early stage (L1), which includes the extraction and identification of the orthographic form of the word, and a later stage (L2), which is solely involved with processing at the phonological and semantic level” (Reingold and Rayner, 745). It was found that the “stimulus quality” of the word (in this instance, how faint or dark it was), affected the first stage but not the second.

These experimental techniques and findings have relevance to such pieces as Jim Andrews’s Stir Fry Texts, which the author identifies as a form of “cut-up” literary work. One section, entitled Blue Hyacinth, presents reading difficulties at the level of content (which changes rapidly with mouseover), but also in terms of colour and contrast, using four shades of blue (from pale to dark), on a black background, thus producing four different negative contrast polarities. (The four shades correspond to four different narratives, which can be viewed separately in their entirety, or cut-up by using mouseover.) Another section, Spas Text, uses shades of white or grey (pale to bright), also on a black background, again differentiating four separate discourses, which again can be subjected to a cut-up process by moving the mouse over the text. Thus both sections use the contrast between faint and brighter text within a line or passage, created by the replacement of elements of one text with elements of another. In this way Andrews uses stimulus quality (found by Reingold and Rayner to influence L1) to delineate aspects of L2 (which source narrative or discourse a given word belongs to, and hence aspects of appropriate semantic processing). In this way, Stir Fry Texts disrupts the normal process of reading at a basic perceptual level, giving semantic significance to features of text which do not normally carry such significance. It also, arguably, plays with the effects of parafoveal processing, that is with the effects which words on the margin of vision have on our processing of the fixated word. In Stir Fry Texts mouse movements create rapid substitutions of phrases within a block of text. As reader, if one always fixates on the current location of the cursor, reading is difficult, both because one is then looking at the cursor not at the text, and because the text beneath the cursor flickers very rapidly. If one does not fixate on the cursor location, then one will often be looking at a (temporarily stable) word, but be aware of surrounding or nearby words and phrases changing. In this way, the work could be said to deliberately invoke a tension between covert and overt attention shifts. The attempt to “manage” the reading process involves the reader in an attempt to use parafoveal processing consciously in combination with fixations, so as to read the word fixated while taking notice of the changes going on elsewhere in the text. 

Andrews’s work also connects with experiments which use “violation of expectation” as an experimental technique. Such experiments are based on a goal-oriented model of reading: to extract a coherent message from a given text. Meaning extraction can become more difficult when sentences contain infrequent words or unusual or ambiguous syntactic constructions. A key result of psycholinguistic reading research has been to document rapid compensatory mechanisms when readers notice problems in meaning extraction. Consider, for example, the sentence “He spread the warm bread with socks”, presented at a rate of one word per second. Up until the last word, readers have no comprehension problems, but the last word violates their expectation and this is noticed within less than 400 ms by the reader’s brain, as indicated by a strong negative deflection in the EEG record obtained during reading (Kutas and Hillyard). When such expectancy violations are presented in text using normal textual layout, regressive eye movements to an earlier part of the current sentence are observed (as demonstrated by Frazier and Rayner).

In Andrews’s work the text substitutions produced by mouse movements introduce phrases from alternative narratives into the short individual narratives. Before such changes, the narratives have a degree of logical coherence. However, the mouse-generated substitutions produce sequences such as “it’s a subtle matter of class, in the bin of a neighbouring premises” (Blue Hyacinth), or “So is resisting in the mirror of course, but having and holding goods and who I am” (Spas Texts). In these instances, “in the mirror” and “who I am” appear in pale grey font, and the rest of the text in off-white font, on a black background. Analogous forms of disruption are a more general feature of much digital poetry, in which the movements of letter or words, and / or other manipulations whether by the programme or the user, frequently create what would be surprising syntactical and other patterns in normal discourse. Is such aesthetic work radically at odds with a goal-oriented model of reading? Certainly its aim is not to maximise ease of simple parsing. Here the two meanings of experimental—empirical and avant-garde—meet at a point of tension. The “disruptions” of reading in digital poetry clearly have affinities with the avant-garde project of transforming or rendering problematic meaning or aesthetic experience itself. In both empirical and avant-garde practice there is a deliberate, strategic creation of difficulty in reading, though with seemingly very different aims.

The disruptive function of avant-garde works is frequently produced by programmatic violations of poetic, linguistic and cognitive-perceptual conventions. These violations include the generation of logical paradox, semantic incongruity, syntactical fragmentation and vacillations between word and image genres that require the reader to shuttle between different perceptual strategies, that is, reading and viewing modes. Whilst the avant-garde experiments are attempts to push further the boundaries imposed by convention on artistic expression, they are also, as discussed earlier, attempts to produce a more critically aware, self-reflexive and in many cases politically radicalised subject.

Psychologists, like digital poets, need to manipulate text for striking visual effects. A surprising font-related result in the field of HEP was recently reported by Miellet et al. who designed a text display that would present the letters in a size that compensated for the reduced visual acuity in peripheral compared to central vision. Given that we can resolve most detail at the point of fixation, and gradually less detail in more peripheral vision, text must be presented in a “butterfly” format. In this format, the directly fixated letter of a given word is printed in normal size, the two adjacent letters are printed in slightly larger size, and more distant letters in ever more increasing sizes, as illustrated in figure 1 (the display was recalculated for each new eye fixation through a rapid, gaze-contingent method).

 

Fig. 2. Graphical depiction of the parafoveal-magnification paradigm, from Miellet et al. Reproduced with the permission of Sage Publications.

The authors expected that readers would be able to process more text at any given fixation, thus leading to larger saccades (eye-movements from one point to another) and somewhat faster reading rates. However, they found that neither reading rates nor saccade lengths differed reliably between conditions. This result was taken to suggest that the crucial cognitive limitation in reading is not our limited perceptual acuity but our limited amount of attention which does not allow us to process more than a small number of letters at a time.

The above figure, with its self-referential allusion to the key elements of the processes being measured (mind and eye), its poetic structure of repetition with difference, and its foregrounding of the materiality of the letter (the pleasing font shaped into slopes and valleys), resembles a concrete poem, such as the well-known “Like Attracts Like” by Emmett Williams:4

Fig. 3. “Like Attracts Like”, by Emmett Williams.

One could also compare it to a “still” of a classic digital poem such as Brian Kim Stefans’s The Dreamlife of Letters (bearing in mind that the image of the parafoveal-magnification paradigm is itself a “still” from a dynamic process):

Fig. 4. Screenshot from The Dreamlife of Letters, by Brian Kim Stefans.

 

5. Letter Forms and Spatial Perception

Another experimental procedure with analogies to digital poetry is the Mental Rotation test, in which participants see a shape or letter which has been rotated at various angles. They are asked to identify whether it is “normal” or a mirror image. Speed and accuracy are assessed as a measure of spatial ability. Examples of this test, using both shapes and letters, can be seen in the Stanford Encylopedia of Philosophy entry on “Mental Imagery”. This experimental procedure may be compared with the “dynamic topology” of John Cayley’s work Lens. Created in the CAVE at Brown University, Lens makes use of 3D-simulation technology to present text in (the illusion of) three rather than two dimensions. 5

 

Fig. 5. Screenshot from maquette of Lens.

 

In an interview, Cayley comments about this work as follows:

Letters are very good at defining space for literate humans. Letter forms give excellent visual clues concerning relative distance. It would require experimentation in perception and cognition to verify this empirically, but my hypothesis is that, because letter shapes are both complex and familiar (to their readers, to the literate), they are highly suitable as reference shapes for spatiality. Unlike abstract shapes, letters possess an intrinsic scale … The implication is that virtual 3D structures made from letter forms will have, as it were, an appreciably enhanced spatial structure for literate readers … it should be possible to “play”—affectively, viscerally—with their form and arrangement in ways that are likely to have aesthetic significance, and some bearing—potentially, ultimately—on literary practice. (Cayley, Interview)

Cayley’s language here – including allusions to “experimentation in perception and cognition” and to possible empirical verification – are indicative of an approach which combines aesthetic and investigative aims. Cayley’s work has been informed by an interest in the phenomenology of perception, which he has brought to his experimental work in the CAVE. This facility is used for purposes such as “scientific visualization, concept visualization, novel artistic concepts, algorithm visualization, behavior simulation, and user interface research” (Brown, website). In an article on Lens, Cayley draws a contrast between his work and the more usual scientific uses of the CAVE:

Virtual reality Caves are used, typically, for scientific visualization—for the graphical representation of complex objects, data sets, mathematical constructs, etc. in an illusionistic but perceptually immersive three-dimensional environment. The underlying assumption is that insights concerning the structure and characteristics of these objects will emerge from the experience of perceiving and interacting with them in 3-D … Due to the nature of the objects concerned—in so far as they are immaterial mathematical constructs and/or only ever accessible to human senses indirectly – any question of a pre-existing phenomenology of these objects is typically bracketed as irrelevant to the scientific purposes they serve. (Cayley, “Lens”)

The use of the CAVE foregrounds potential connections between scientific and aesthetic projects, but also the difference, mentioned above in relation to Psycholinguistics, between the artist’s interest in cultural particularity, and the scientific drive towards universal laws. Cayley goes on to explain further his understanding of the cultural particularity which he terms the “phenomenology of an object”:

I mean simply the significance and affect that human subjects ascribe to the object – its cultural form and meaning as attested by a person or by persons who simply … encounter and experience the object … I want to contrast its underlying role in the scientific use of the Cave as opposed to its potential use for artistic practice, especially literary practice. In order to generate aesthetic significance and affect, an artist will, typically, be required to engage with the pre-existing phenomenology of whatever objects her work evokes and inflects … unlike a scientific visualizer … I cannot bracket the phenomenology of the objects I project. (Cayley, “Lens”)

In this article Cayley describes an “experimental” approach which leads to the formulation of a hypothesis (rather than deriving from one, as in the standard HEP approach):

This powerful perceptual experience is demonstrable and repeatable, despite its artificiality and strangeness. The question arises, why should this phenomenon [the spatial illusion created by letter size in the CAVE, in which small letters rendered over large ones nevertheless appear to be further away] be so immediate and effective? … [W]hy doesn’t the linguistic materiality of the graphic forms and structures run counter to their visuality, counter to whatever illusion of space may or may not be generated? At this point I began to formulate a hypothesis: literal forms are highly effective for delineating space in immersive virtual environments. (Cayley, “Lens”)

Cayley’s own discussion of Lens focuses on features of perceptual habit and expectation as key issues in relation to poetic work. In one sense, Cayley’s work could be seen as exploratory attempts to use the CAVE for research in spatial perception and its implications for “user interfaces”; as a way of generating hypotheses which might then be subject to testing according to a standard HEP experimental method, as described above in Section 2. Cayley himself does not seem hostile to such a possibility. Yet his works clearly are, in intent and effect, aesthetic artefacts, and his comments quoted above concerning “a shared and contested phenomenology of language” register a crucial distinction. Here the art vs. science (or creative practice vs. HEP) binary is not constructed along the familiar lines of the subjective and emotional vs. the objective and rationalistic. Rather, the crucial feature of the aesthetic is that it cannot “bracket” the “phenomenology of the object”, which Cayley defines as “the significance and affect that human subjects ascribe to the object” (Cayley, “Lens”). This phenomenology would be a shared but individually variable set of assumptions and habits of perception arising from how we learn to read, habits of reading, the cultural role of reading, the locations and forms in which text is normally encountered in a given society, the cultural value, shared meanings and personal associations attached to text, and to particular sorts of text (such as poems). Cayley emphasizes in particular that our learned, culturally-shaped perception of text is closely tied to (seemingly two-dimensional) “opaque and resistant” surfaces (Cayley, “Lens”). 

The question of bracketing leads us into some of the cross-currents in the interaction of aesthetic and scientific modes of thought. Cayley’s usage of this term, in the context of phenomenology, presumably draws on Husserl’s method of epoché, of which The Stanford Encyclopedia of Philosophy gives the following succinct account:

Husserl’s transcendental turn also involved his discovery of the method of epoché (from the Greek skeptics’ notion of abstaining from belief). We are to practice phenomenology, Husserl proposed, by “bracketing” the question of the existence of the natural world around us. We thereby turn our attention, in reflection, to the structure of our own conscious experience. (Smith) 

This is rather different from Cayley’s allusion to the “bracketing”, in scientific uses of the CAVE, of the “cultural form and meaning [of an object] as attested by a person or persons who simply … encounter and experience the object” (2). Indeed, the two forms of “bracketing” might be seen as opposites. Husserl proposes bracketing the existence of the object in favour of attention to the experience of the observer, whereas scientific applications of the CAVE bracket the cultural influences on the observer’s experience in favour of an attempt to determine the “structure and characteristics of the object” (Cayley 2). Cayley’s point is perhaps precisely that these scientific uses presuppose the objective existence and characteristics of the object (leaving aside for the present the potential ironies arising from the fact that the “objects” in question may be virtual or conceptual, being “data sets” or “mathematical constructs”). However, Husserl’s method is not merely one which directs attention to the subjective: it rather attempts a form of reconciliation of the objective and subjective, in the form of an objective study of subjective experience. To quote The Stanford Encyclopedia again:

For Husserl, then, phenomenology integrates a kind of psychology with a kind of logic. It develops a descriptive or analytic psychology in that it describes and analyzes types of subjective mental activity or experience, in short, acts of consciousness. Yet it develops a kind of logic – a theory of meaning (today we say logical semantics) – in that it describes and analyzes objective contents of consciousness: ideas, concepts, images, propositions, in short, ideal meanings of various types that serve as intentional contents, or noematic meanings, of various types of experience. (Smith) 

These “objective contents of consciousness” might correspond to Cayley’s “cultural form and meaning”, so that Cayley is implicitly contrasting a “scientific” bracketing of the phenomenology of the object (in favour of attention to its “structure and characteristics”) with a phenomenological (and aesthetic) bracketing of the text as (virtual) “object” in favour of attention to the (shared) experience of the readers or interactors.

In this distinction, one would perhaps expect empirically-based Human Experimental Psychology to be on the side of the scientific. Indeed, it is true that, in so far as HEP is aimed at isolating generalisable variables in perceptual and cognitive processes, and supporting hypotheses about general laws in respect to brain-body functioning, it precisely requires the bracketing of cultural and individual variation and of assumptions of meaning and value. On the other hand, the specific culturally-learned perceptions to which Cayley refers may be amenable to study by experimental means. HEP would share with phenomenology the aim of studying subjective phenomena in objective terms, but would be less concerned with the “objective contents of consciousness” than with the processes of perception and cognition by which those contents are handled.

Terry Eagleton argues that what Husserl proposes as a new “universal science of subjectivity” is in fact a reinventing of the aesthetic itself, in that Husserl’s imperative to “consider the surrounding life-world concretely, in its neglected relativity” in effect describes aesthetic experience (Eagleton 18). But he also notes Husserl’s attempt to reconcile this with scientific thought: “it turns out conveniently enough that the life-world discloses just the same structures that scientific thought proposes in its construction of an objective reality” (18). It is clearly beyond the scope of the present discussion to resolve any of the issues in the philosophy of science which arise from these reflections. However, such reflections serve to illustrate some of the consequences when creative practice, and its theorisation (as in Cayley’s CAVE practice and his exposition of that practice), share technological means and resources with scientific research. Digital poets and artists are drawn into an arena in which scientific and aesthetic questions follow some of the same trajectories, going beyond simplistic antitheses between the objective and subjective, even while tensions within and between science and creative practice are revisited. 

6. Interactivity and the Construction of the Subject in HEP and Electronic Literature

We will now discuss in more detail Tower, by Simon Biggs and Mark Shovman, as a case study for the emerging dialogue and relationship between digital artists and psychologists. Tower, which was commissioned for the Poetry Beyond Text project, was the product of a collaboration between a digital artist (Biggs) and a psychologist / computer programmer (Shovman). It is an immersive 3D textual environment combining visualisation, speech recognition and predictive text algorithms. The following is Biggs’s own description of the work:

This work combines full 3D immersive visualisation, speech recognition and predictive text algorithms to create an immersive 3D textual environment that interacts with and forms, over time, from the utterances made within it, forming a heuristic dialogue of probabilities. The work engages first, second and third-person inter-subjectivity. Viewers occupy one of three roles: a spectator of others interacting with the work, one of several interactors within the 3D interactive environment or a central interactor who is fully immersed in the 3D virtual environment. The first-person interactor experiences the environment wearing a VR head-display. Located at the vertiginous pinnacle of a virtual spiral word structure, when they speak the words appear to float from their mouth and join the spiralling history of previously spoken words. As the word emerges other words spring from the spoken word which are those predicted to be spoken next, based on the statistical frequency of word sequences in a stored textual corpus. Wearing 3D spectacles within a panoramic immersive 3D projection, the second-person interactors, co-located with the first-person interactor, view them at the top of the spiral-word structure, words appearing from the first-person interactor’s mouth as they speak and the spiral gradually growing. The spiral resembles a Tower of Babel composed of words. The third-person observers stand outside the interactive zone without 3D spectacles. (Biggs, Description of Tower)

 

Fig. 6 Composite Image of Tower in the HIVE at Abertay University

 

The version exhibited at Abertay University in March 2011, as part of the concluding exhibition of the Poetry Beyond Text project, used as corpus a combination of James Joyce’s Ulysses and Homer’s Odyssey, so that the work’s predictions of words were based on the sequential occurrence of words in those literary work. The HIVE at Abertay University in Dundee (a single screen 3D simulator rather than an immersive “room”, but in other respects comparable to the CAVE) is used, for example, to train police offers in shooting scenarios. It is also used to present complex numerical / statistical data from scientific experiments in the form of a 3D simulation, in order to aid in the detection of patterns in these data (a set of points which “look” random in 2D may reveal a potential “shape” when rotated in 3D). Here we see a certain convergence at a very basic level. Clearly the creation and detection of patterns is fundamental to many forms of human creativity, including scientific work and artistic practice. In fact, the human brain has been characterized as a pattern detector because we learn by discovering regularities in our environment and our nervous system changes its structure as a result of such co-occurrences (Hebbian learning). This analogy raises issues in the philosophy of science and aesthetics, including those already alluded to in the discussion of “bracketing”: issues about the relationship between creating and perceiving patterns; about what is given to us from “the world”, what is imagined by us, and what is a function of our perception of the world. These are, of course, classic questions in Romantic theories of perception, such as those of Wordsworth and Coleridge, articulated in these lines from “Tintern Abbey”: “of all the mighty world / Of eye, and ear, –both what they half create, / And what perceive”. Pattern recognition also plays a key role in literary analysis. For example recognising repetitions, rhymes and metrics is an important part of reading poetry. In the space of the screen, moreover, these elements can be continually shifting and morphing into new patterns, creating a situation where reading is less the extraction of a stable content generated by fixed form, than a continual process of adjustment to new patterns and an attempt to see within these a larger, temporal pattern. Code itself, the script in which digital literature is programmed, is fundamentally a pattern system rather than a language as such; code is a system in which the different combinations of 1s and 0s generate visual audio and linguistic forms.

We have proposed the idea of the “experimental” as a possible link between digital creative practice and HEP. Biggs does see an element of the “experimental” in his work – not in the conventional artistic sense of that term, which merely privileges novelty, but in the sense of accepted risk and progressive learning:

I use the term “experimental” to describe my work in so far as each work is an experiment in apprehension, itself made up of smaller experiments. Some works are more experimental than others. As the works are experimental they sometimes fail in their objectives but failure informs further iteration. (Biggs, private e-mail)

The term “apprehension” here suggests the (en)active model of perception, apprehension being perception conceived of as an active seizing hold. The configuration of Tower, specifically the degree of equality in activity which it performs between work and interactor, might be seen as a polemical revisioning of the process of scientific experiment in the light of postmodern and other problematisations of objectivity and the split between neutral observer and hypostatized “subject” of observation.

Here the concept of “interactivity” (a crucial though problematic term in relation to electronic literature) comes into play, raising in particular questions of constraint, choice, empowerment and the ideological construction of the human subject. Michelle Kendrick argues that “the hypertext rhetorics of interactive reading” parallel “claims in postindustrial capitalism that defined democratic agency as choice among products” (248), so that claims for the empowerment of the reader (made by some hypertext theorists) are politically complicit with consumerism and global capitalism:

Thus we have the “promise” of hypertext, decoupled from the author, divorced from content and instead driven by a rhetorics that celebrate the “empowered” reader, free to choose from a seemingly infinite array of options. The fantasy of cyberspace is the fantasy of infinite exchange in a realm irrevocably encoded as capitalist. (Kendrick 249)

While her comments are specifically based around hypertext, and the model of clicking as choice, comparable issues may be raised about other works of electronic literature in which the physical process of choosing options comes to the fore. In a work such as Tower, the viewers and interactors do not choose options, but respond to a process which responds to them. There is an element of simulation of artificial intelligence. Espen Aarseth notes some of the implications of mutuality between humans and machines: 

Andrew Lippman … sees [interactivity] as “mutual and simultaneous activity on the part of both participants, usually working towards some goal, but not necessarily” … This is a daring definition, as it implies a functional equality between the interacting agents and a relationship of some sort. Of course, everything hinges on the word “mutual”. Defined in this way, interactivity between human and machine can take place only if the machine is somehow aware of the situation. This of course conjures up all the problematic issues of artificial intelligence. (50–1)

A crucial question would seem to be whether the form of interactivity offered by Tower avoids the critique of interactivity as commodification of individual choice. “Choice” does not seem a particularly relevant term for Tower. On the other hand, there is arguably a commodification of interactivity itself in progress in developed societies, as consumer products are marketed on the basis of their “interactive” capabilities. Biggs’s response to this is to construct interactivity and creativity as properties of groups or communities, rather than of individual works or artists.6

Discussions of interactivity tend to focus on the ability of the user to change the work. What of the ability of the work to change the user? Psychological experiments seek to avoid changing the mind of the participant during the experiment (except in certain specific, defined ways where such a process of change is the process being studied): for example, they would try to avoid “contamination” of results by previous experience through systematic allocation of experiences to participants, so that each participant experiences a certain stimulus (say, a particular poem) subsequent to a different previous stimulus (the so-called counterbalancing technique). The overt aim of such experiments is to prevent similar and systematic changes of all the participants, except temporarily and in terms of a specific dependent variable. HEP tends to hypostatize the human object of study (the “subject” or “participant”) with reference to the “objective” technologies of measurement, such as eye tracking or fMRI (functional magnetic resonance imaging) scanning used to measure him or her. Machine–user interactions, if these involve the machine affecting the user, are likely to be regarded as problems of interference or noise. Examples would be the effect of noise in MRI scanners, or the physical immobility required during eye-tracking. The exception would be an experiment in which such an effect constituted the independent variable—for example if one was studying the effect of the MRI noise on patients’ well-being. Hayles and others have developed the idea of the human-machine hybrid or cyborg. Might the situation of a HEP experiment constitute such a hybrid? Rather than seeing it as an “objective” piece of technology used to “study” a human subject, we might see it as constructing a hybrid human-machine subject. Kendrick applies the Latourian concept of the hybrid which, “within the myths of modernity” “confound[s] the desired division between the natural and the technological” (234):

Such hybrids necessitate discourses of purification – cultural narratives that redistinguish nature from technology by camouflaging hybrids, while still permitting the miscegenation of these constructs to continue. This narrative sleight of hand allows a preexisting subject to create, use, and remediate technology while denying or repressing the fact that his or her existence cannot be isolated from a host of technological interventions. (Kendrick 234)

Questions about the conception of the subject in HEP of course raise issues, too large to be considered here, about the relationships between brain and mind and between mind and sense of identity. Kendrick describes the allegedly “liberated” reader of hypertext, in Latourian terms, as a “subject … both created by technology and prior to technological intervention” (234). Comparably, the participant in a psychological experiment would conventionally be constructed as a product of nature (evolution, biology, brain structure etc), “prior to technological intervention”, even if inflected by culture. Yet increasingly, as the technologies of experiment cross over with those of communication, entertainment, business and other activities, the experience of the participant in an experiment will be mediated by their prior experience of technology, such that their interaction with the experimental apparatus may take place on a continuum with their regular interaction with digital and other technologies. This is evident, for example, in the instance of the dual experimental and commercial use of RSVP discussed earlier (for studying reading processes and for use in hand-held reading devices). 

Biggs’s comments on the interactor’s experience of Tower evoke something resembling an opposite to the hypostatization of the human subject that we have suggested occurs in HEP:

My hope is that a person experiencing Tower (or any artwork or, for that matter, any experience) will be enabled to reflect upon the inter relations of the things they are experiencing and their own contingency as part of that set of things. This could potentially, and ideally, result in a fugue state where the person is given cause to consider their condition at that moment as a non unitary thing and as a more or less temporary set of relations of things. (Biggs, private e-mail)

Alluding here to the form of amnesia known as “dissociative fugue”, which is specifically associated with loss of identity or multiple identities, Biggs applies to aesthetic experience an effect normally associated with traumatic life events. Whereas HEP experiments are intended to provide the observer or experimenter with data (as far as possible “objective”) about the perceptual and cognitive processes of a “subject” (an individual brain-mind-body possessed of distinct capabilities, potentialities, and cultural context), Tower is intended to provide an individual with an experience which would subjectively call into question the assumption of the unity of the subject, ontologically and through time.

Biggs’s view of the vexed concept of interactivity would certainly also push us in the direction of a revisioning of ideas of the “objective” study of cognition:

I reserve the term “interactive” for those artworks that change themselves, into another state or form, generated by the work (or the code within it) through interaction with another agent. The theory of computability (Goedel) and third-order cybernetics (Maturana and Varela) describe this process. (Biggs, private e-mail)7 

Biggs’s allusion here to third-order cybernetics bears directly on the question of scientific procedures, since second-order cybernetics questioned the objective status of observed reality, while third-order cybernetics turns such scepticism onto the assumed coherence of the observer. As summarised by Philip Boxer and Vincent Kenny:

Second-order cybernetics was born … by generating a domain of discourse about observers, and the ways in which they brought forth the apparently objective realities which they enacted. Heinz von Foerster and others who were involved in the generation of 2°C were keen to problematise the assumption that there was an objectively existing reality which was “there” independent of any observer. (Boxer and Kenny 2)

Of particular relevance to Tower may be the interest of third-order cybernetics in multiple observers:

To parenthesise the observer position itself, we need a 3°C in order to properly examine “Who” it is that “holds” these purposes, and indeed to examine the “purpose of purpose”, or in other words, the “Why” of the observer. We are particularly interested to discern the ways in which the multiple meanings of multiple observers—manifested in differentiated speaking and listening positions—are coordinated and organized in terms of a 3° Cybernetics diachronically in relation to each other; and synchronically in relation to an ideal. (Boxer and Kenny 6–7)

Tower performs or dramatizes “differentiated speaking and listening positions”. The experience of the first-person interactor (wearing the VR helmet) might be described using the familiar literary term “defamiliarization”, but what is defamiliarized is not the referent of language, as in Shklovsky’s theory, but its materiality, which is “made strange” by assuming an existence in space. The interactor may experience some vertigo, as they seem to stand at the top of a cone of spiralling words. He or she may wonder whether to interpret this structure symbolically, or merely to accept it as a sensory “experience”. VR almost by definition creates what Biggs terms “a more or less temporary set of relations of things” (quoted above), in that it places us in a perceptual environment which is discontinuous with that which we normally inhabit.

As already noted, Tower is a product of more than one discipline. The following are some of Biggs’s comments on the collaborative process:

Mark and I are both programmers, although Mark is a more experienced and better educated programmer than I have ever been. Nevertheless, we share an understanding of what coding means, what the processes involve, what the key concepts are and why they can be intellectually exciting. We share a passion about computers and formal languages and thus share that language to an extent. (Biggs, e-mail) 

This echoes, with a difference, the observation made by many authors that passion for language(s) itself / themselves is crucial to literary achievement (though Biggs does not regard himself as a poet or author, but as a digital artist who uses words). Shovman’s previous work in the HIVE had included creating programmes for modelling complex sets of scientific data, and this experience fed into the creative process for Tower:

For me the value in working with Mark that was most productive was gaining a better insight into how data can be represented and modelled and how these processes can impact upon our appreciation of data-sets. I also found it inspiring to see how these processes could be normalised within software. These insights directly influenced how the geometric structure of Tower turned out. (Biggs, e-mail)

Again, this comment is, in some ways, entirely in key with the general view expressed by poets and artists, that originality emerges from an engagement with the medium. But it also shows how the sharing across disciplines of the technology that enables this particular medium can generate interaction between science and art at the level of technical detail as well as that of broader inspiration: how imaginative responses to an artwork can converge with the “appreciation of data sets”.

7. Conclusion: Models of the Relationship between Digital Creative Practice and HEP

How might we configure this putative relation between experimental psychology and digital creative practice – a relation which can clearly take various forms? One possibility (though one loaded with complex overdeterminations) would be in terms of the relation between a (rationalistic, scientistic, progressive) modernity and an avant-garde practice which both seeks to reunite art with social practice (in this instance via technologically-supported convergence with the mass popularity of digital media, such as games) and auto-critiques its own practice. One could note in this context Richard Murphy’s questions about expressionism, as a possible instance of avant-garde practice: 

to what degree does expressionism fulfill the avant-garde’s role of producing a fundamental rethinking of the artist’s social practice, together with a full-scale interrogation of the social and institutional conditions of art? To what extent does it remain caught within modernism’s predilection for aesthetic autonomy and its drive for purely technical and formal progress? (4)

Could we say that HEP is (in this sense) modernist (technocratic and functionalist), whereas digital literature is “avant-garde” (interrogating techniques of mediation)? If that were the case, then the problematic relations of interchange and tension between modernism and the avant-garde would offer a suggestive model for the relationship between the two.

Biggs is sceptical about the association which is sometimes made between creative practice in contemporary emergent media and the idea of the “avant-garde”; he has argued that the absence of a homogenous mainstream or bourgeois culture in contemporary society (due to the rise of globalisation and multiculturalism), precludes the existence of an effective avant-garde, a position in accord with Peter Bürger’s arguments for the historical nature of the avant-garde project (Biggs, ‘Multimedia’).8 However, certain aspects of the avant-garde project – notably the aim of a reintegration of art and life, and a wish for transformations in our experience or understanding of our own subjectivity – have a relevance to Biggs’s work and other works in emergent media. Digital art practice has a transformative potential in relation to ordinary life, because it can intervene in activities and processes (gaming, the internet, automated control systems, cybernetics, HCI) which are becoming increasingly dominant as forms in which “life” takes place in developed societies, and because the boundaries between art and other practices using this range of technologies are increasingly blurred. This is reflected in Biggs’s comment on the value underpinning Tower:

Perhaps the primary value, then, is its potential for disruption of the self model and subsequent impact on the perception of other things. (Biggs, private e-mail)

This implies an ambition for artistic practice to have a transformative effect on “ordinary life”.

Another configuration for the relation between experimental psychology and digital creative practice, more binary and oppositional in its implications, would be in terms of a contrast between scientific realism as manifest in the methods of human experimental psychology as described above, and the more anti-realist position of both second- and third-order cybernetics. Second-order cybernetics crucially introduces the role of the observer in bringing “forth the apparently objective realities which they enacted” (Boxer and Kenny 2). While the particular debt to certain ideas in third-order cybernetics may be specific to Biggs, the interest in cybernetics itself, and in “the multiple meanings of multiple observers – manifested in differentiated speaking and listening positions” is widespread in electronic literature. Indeed, that phrase could also be applied to much of the “alternative” and avant-garde forms of print poetry and postmodernist fiction on which electronic literature draws. HEP in general seeks to bracket or eliminate observer effects which might be held to bring “forth apparently objective realities”; it is more interested in maintaining a pursuit of objective realities. Here there may be a radical incommensurability between the two disciplines or practices. 

In our introduction we posed the question whether the engagement between psychological experiment and creative practice should be seen as local and transitory or as symptomatic of more significant trends. While it may be premature to choose a model for such an emerging and developing relationship, we would suggest on the basis of our own work and observations that the depth and complexity of this engagement is likely to increase.

 

Works Cited

Aarseth, Espen J. Cybertext: Perspectives on Ergodic Literature. Baltimore and London: Johns Hopkins UP, 1997.

Andrew, Jim. Stir Fry Texts. 1999- . Electronic Literature Collection. Ed. N. Katherine Hayles, Nick Montfort, Scott Rettberg, and Stephanie Strickland. 2006. Vol. 1. 26 Sept. 2012 http://collection.eliterature.org/1/works/andrews__stir_fry_texts.html 

Back, Maribeth, Jonathan Cohen, Steve Harrison, and Scott Minneman. “Speeder Reader: An Experiment in the Future of Reading.” Computers and Graphics 29 (2002): 623–27.

Balsamo, Anne, Matt Gorbet, Steve Harrison, and Scott Minneman. “The Methods of our Madness: Research on Experimental Documents.” Proceedings of CHI 2000: The Future is Here. 26 Sept. 2012 http://portal.acm.org/citation.cfm?id=633411&dl=ACM&coll=DL&CFID=27701994&CFTOKEN=70533938

Biggs, Simon. Messages to Andrew M. Roberts. May-June 2011. E-Mail.

———. “Multimedia, Multiculturalism, Language and the Avantgarde: Notes and Observations from E-Poetry 2007.” Institute for Distributed Creativity (iDC Listserv). 24 May 2007. 26 Sept. 2012 https://lists.thing.net/pipermail/idc/2007-May/002533.html

———. “Tower.” Poetry Beyond Text. Catalogue. 2011. 

———. “On Navigation and Interactivity.” p0es1s: Ästhetik digitaler Poesie/The

Aesthetics of Digital Poetry. Ed. Friedrich W. Block, Christiane Heibach and

Karin Wenz. Ostfildern-Ruit: Hatje Cantz, 2004. 181-192.

Biggs, Simon, and Penny Travlou. “Distributed Authorship and Creative Communities.”

Dichtung Digital 41 (2012). 26 Sept. 2012 http://www.dichtung-digital.org/2012/41/biggs-travlou.htm

Boxer, Philip, and Vincent Kenny. “Lacan and Maturana: Constructivist Origins for a 3° Cybernetics.” Communication and Cognition 25.1 (1992): 3-100.

Brown University Computer Graphics Group. Cave: Project Overview. 26 Sept. 2012 http://graphics.cs.brown.edu/research/cave/home.html

Cayley, John. “An Interview with John Cayley on Torus.” Interview with Rita Raley. 2006. 26 Sept. 2012 http://iowareview.uiowa.edu/TIRW/TIRW_Archive/september06/cayley/cayley.html#

———. ”Lens: The Practice and Poetics of Writing in Immersive VR (A Case Study with Maquette).” Leonardo Electronic Almanac 14.5-6 (2006). http://www.leoalmanac.org/leonardo-electronic-almanac-volume-14-no-5-6-august-september-2006/

———. Lens. (maquette for web). March 2006. 26 Sept. 2012 http://programmatology.shadoof.net/index.php?p=works/lens/lens.html

Cocklin, Thomas G., Nicklas J. Ward. Hsuan-Chih Chen, and James F. Juola. Factors Influencing Readability of Rapidly Presented Text Segments. Memory and Cognition 12.5 (1984): 431-442.

Eagleton, Terry. The Ideology of the Aesthetic. Oxford: Blackwell, 1990.

Frazier, Lyn, and Keith Rayner. “Making and Correcting Errors during Sentence Comprehension: Eye Movements in the Analysis of Structurally Ambiguous Sentences.” Cognitive Psychology 14.2 (1982): 178-210.

Hansen, Mark B. N. New Philosophy for New Media. Cambridge, MA: MIT P, 2004.

Hayles, N. Katherine. Electronic Literature: New Horizons for the Literary. Notre Dame, IN: U of Notre Dame P, 2008.

Jenkins, Henry. “Confessions of an Aca-Fan: The Official Weblog of Henry Jenkins.“ 26 Sept. 2012 http://www.henryjenkins.org/2006/06/welcome_to_convergence_culture.html 

Kendrick, Michelle. “Interactive Technology and the Remediation of the Subject.” Configurations 9.2 (2001): 231–51. 

Klastrup, Lisbeth. “Paradigms of Interaction: Conceptions and Misconceptions of the Field Today.” Dichtung Digital 30.4 (2003). 26 Sept. 2012 www.dichtung-digital.org//2003/4-klastrup.htm

Kutas, Marta, and Steven A. Hillyard. “Reading Senseless Sentences: Brain Potentials Reflect Semantic Incongruity.” Science 207 (1980): 203-5.

Juola, James. F, Nicklas J. Ward, and Timothy McNamara. “Visual Search and Reading of Rapid Serial Presentation of Letter Strings, Words, and Text.” Journal of Experimental Psychology: General. 111.2 (1982): 208-27.

MacLin, Kimberly M., and Robert L. Solso. Experimental Psychology: A Case Approach. 8th ed. Zug: Pearson, 2009.

Miellet, Sébastian, Patrick J. O’Donnell, and Sara C. Sereno. “Parafoveal Magnification: Visual Acuity does not Modulate the Perceptual Span in Reading.” Psychological Science 20.6 (2009): 721-28.

Murphy, Richard. Theorizing the Avant-Garde: Modernism, Expressionism, and the Problem of Postmodernity. Cambridge: Cambridge UP, 1999. 

Oequist, Gustav, and Goldstein, Mikael. “Towards an Improved Readability on Mobile Devices: Evaluating Adaptive Rapid Serial Visual Presentation.” Interacting with Computers, 15.4 (2003) 539-58.

Potter, Mary. C, Judith F. Kroll, and Carol Harris. Comprehension and Memory in Rapid Sequential Reading. Attention and Performance. Vol. VIII. Ed. Raymond S. Nickerson. Hillsdale, NJ: Erlbaum, 1980. 98-118. 

Pressman, Jessica. “The Strategy of Digital Modernism: Young-Hae Chang Heavy Industries’s Dakota.” MFS 54.2 (2008): 302-26. 

Reingold, Eyal M, and Keith Rayner. “Examining the Word Identification Stages Hypothesized by the E-Z Reader Model. Psychological Science 17.9 (2006): 742-46. 

Schaffner, Anna Katharina. “From Concrete to Digital: The Reconceptualisation of Poetic Space. Beyond the Screen: Transformations of Literary Structures, Interfaces and Genres. Ed. Peter Gendolla and Jörgen Schäfer. Bielefeld: Transcript, 2010. 179–97.

Smith, David W. “Phenomenology.” The Stanford Encyclopedia of Philosophy. Ed. Edward N. Zalta. Summer 2009 Edition. 26 Sept. 2012 http://plato.stanford.edu/archives/sum2009/entries/phenomenology/

Stefans, Brian Kim. The Dreamlife of Letters. Electronic Literature Collection. Ed. N. Katherine Hayles, Nick Montfort, Scott Rettberg, and Stephanie Strickland. 2006. Vol. 1. 26 Sept. 2012 http://collection.eliterature.org/1/works/stefans__the_dreamlife_of_letters.html

Thomas, Nigel J.T. “Mental Imagery.” The Stanford Encyclopedia of Philosophy. Ed. Edward N. Zalta. Fall 2010 Edition. 26 Sept. 2012 http://plato.stanford.edu/archives/fall2010/entries/mental-imagery/

Tinker, Miles A. Legibility of Print. Iowa: Iowa State UP, 1963. 

Young-Hae Chang Heavy Industries. “Beckett’s Bounce.” 2002. 26 Sept. 2012 http://www.yhchang.com/BECKETTS_BOUNCE.html

———. Dakota. 2001. 26 Sept 2012 http://www.yhchang.com/DAKOTA.html

———. Lotus Blossom. 2002. Sept 26 2012 http://www.yhchang.com/LOTUS_BLOSSOM.html

 

 

 

Footnotes
  1. Project Team Members: Professor Andrew Michael Roberts (English, University of Dundee), Dr Anna Katharina Schaffner (Comparative Literature, University of Kent), Professor Martin Fischer (Psychology, University of Dundee), Dr Ulrich Weger (Psychology, University of Kent), Ms Mary Modeen (Fine Art, University of Dundee), Dr Lisa Otty (English, University of Dundee), Dr Kim Knowles (Comparative Literature, University of Kent). For more information, please see www.poetrybeyondtext.org  back
  2. The first of these options is not accorded much respectability in critical thought, but is often implicit in general discussions (such as in reviews).  back
  3. There are parallels with the avant-garde film tradition: the rapidly flashing text, and the problems seemingly posed by it, evoke the “flicker’ films of the 1960s and 70s, such as those made by the Fluxus artist Paul Sharits. For example, in his 1966 film Word Movie single words appear successively at a rate of one frame per word, while each word shares a letter with that which precedes it, providing an impression of continuity in an otherwise random reading exercise.  back
  4. On concrete poetry as precursor to digital poetry see Schaffner.  back
  5. The CAVE at Brown University is an eight-foot cubicle in which high-resolution stereo graphics are projected onto three walls and the floor to create an immersive virtual reality experience. High-end workstations generate the 3D virtual world and create the sounds of the environment. Special hardware and software keep track of the positions and movements of a person entering that virtual environment, changing the images in the cave in a way that allows the visitor to feel immersed in the virtual space. (Brown, website)  back
  6. “Whilst creativity is often perceived as the product of the individual artist, or creative ensemble, it can also be considered an emergent phenomenon of communities, driving change and facilitating individual or ensemble creativity. Creativity can be a performative activity released when engaged through and by a community and understood as a process of interaction.’ (Biggs and Travlou, abstract). The original of this article is to be published in the proceedings of the 2010 Society for Science, Literature and the Arts European conference (Riga) and a revised version is published in the current issue of Dichtung Digital.  back
  7. For a fuller discussion of interactivity, cf. (Biggs, “On Navigation and Interactivity”).  back
  8. Much of the initial discussion in the E-poetry 2007 conference focused on the relation between the avantgarde and digital poetics. The premise was that digital poetics represents a new avantgarde and that from this it follows that digital poetics is a good thing. That the avantgarde can only exist in relation to a largely homogenous society is overlooked in this argument. Contemporary heterogeneous social environments do not offer the easy target of a mainstream or bourgeoisie against which an avantgarde can differentiate itself.’ (Biggs, “Multimedia’)  back

0 Comments

 

Leave a comment

Please fill out all the fields.
Your email adress will not be published or distributed.