John Cayley: Overboard

The above paragraph is the brief explanatory text which, currently, I have associated with the piece overboard in its most of its various forms of distribution. Roberto Simanowski has asked me to go into more detail to describe and explain the piece and its 'programming,' programming, that is, broadly conceived as both algorithmic coding and as what might be termed 'rhetorical configuration.'

For some time, my digitally mediated literary work, my literal art (1), has been involved, in terms of one of its most obvious formalisms, with transliteral morphing from one given text - transcribed in machine-encoded alphabetic script - to another. I have developed various algorithms, based on literal identities and their sequenced transposition, to perform these morphs in software-driven systems (www.shadoof.net/in). My intention has been, in part, to interrogate certain relationships between the granular or atomic structures of alphabetically transcribed language and the critically or interpretatively discoverable rhetorical and aesthetic effects of literature.

The algorithms are not particularly 'complicated,' 'powerful,' or 'intelligent,' even given the restricted metaphorical tenor of these apparently strong adjectives when they appear in the language of programmers and students of technoculture. I am just enough of a programmer to know the limitations of what I am doing in these terms. My investment lies in the belief that relatively simple algorithmic manipulation of basic low-level, para-semantic linguistic systems is able, significantly, to yield rhetorical and, indeed, aesthetic effects which can be correlated with their programmatological generators. For example, I would claim that the iterative transliteral morphs between related texts - texts that might be seen, for example, as rewrites in differing styles - will reveal abstracted underlying structures supporting and articulating the 'higher-level' relationships between the texts.

overboard began with an inflection of my original morphing algorithm and its effects. I became interested in letter substitution that would be in some sense 'minimal,' the substitution of one letter for another that is as similar as possible - similar, that is, according to any culturally perceivable scale of characteristics defining a letter's 'identity': its form, the linguistic sound(s) to which it refers, its position in any shared literal schema (such as the alphabet itself and its traditional ordering), and so forth. The particular transliteral process that developed from this inflection of my original morphing algorithm also became associated with the possibility of a less usual performance of poetics, one that was 'ambient,' in the sense of ambient music (2). I wanted to make a piece that was unambiguously literary but that might perhaps hang on wall-mounted flat screen, like a kinetic literary painting. The viewer or reader would see a textual image with a recognizable underlying form, but this would change constantly by way of its minimal letter substitutions, ideally such that the changes would be barely perceptible. The piece would seem not to change and yet always to be different, whenever it was given any attention.

In its present form, the constant transitions in the text of overboard are too obvious and too varied to fulfil that particular vision. Its primary instantiation as monitor-based work has necessitated a modulated presentation. A wall-mounted version could still, of course, be made, slowed down and retuned to recover that earlier intention, but that will have to wait for some future project.

In the meantime, apart from subliminal, ambient transliteral change itself, overboard became concerned with legibility, literal legibility, I'm tempted to say. The screen-based work produces, successively a set of three metaphorically implicated textual states, that relate, in a fairly obvious way, to the representational sense of the underlying piece of natural-language writing. Thus, the formal procedures applied to the literal structure of the underlying text, mirror its semantic representations. The text itself, extracted and adapted from the account of an incident during the Mayflower crossing in Governor Bradford's Of Plymouth Plantation, tells the story of a man who was swept overboard and later hauled back on board the ship alive. The literal text enacts processes that I have come to think of as floating, drowning and surfacing, in terms of literal legibility.

I will not quote the entire text here, despite its brevity, since as I say, legibility is at issue for its appreciation and although the text is not concealed, I am unsure as to whether a reader benefits aesthetically in appoaching the actual performance of the work, if they either know or do not know what the underlying text spells out:

  ... in a mighty storm
  a man came above board
  and was thrown into the sea

  but he caught hold of the halyards
  which hung over board
  and held his hold
  though he was many fathoms under water

  till he was hauled up
  to the brim of the water ...

The underlying text is set out with line and stanza breaks in the manner of poetic verse. Each of the four verses in the complete text may, independently, be in any one of the three states: floating, drowning or surfacing. As overboard opens, in its current version, the display space is black and all verses are placed in the 'surfacing' state. This means that the underlying text is scanned by the surfacing algorithms and its constituent letters gradually appear on the 'surface' of the display space in a quasi random sequence, white on black, the letters slowly accumulate to constitute the textual field. The text is set in a monospaced (or fixed-width) font so that letters in the notional textual grid have regular fixed positions. On the left-hand side of the screen a visual correlative of the text and its textual processes is displayed, with small images - fragments of a photograph of the sea's surface - corresponding, in position and identity, with the letters of the text. This visual correlative will be discussed briefly later.

overboard's scheme of minimally distinct alternate letters also operates during the surfacing algorithm. During surfacing this means that, at a particular letter position, if the space is blank then either the 'natural' letter (the letter in this position in the underlying text) or its alternate may appear (3). However, when surfacing, if the natural letter appears it will persist, whereas its alternate letter will eventually change to the target letter over time. So long as the surfacing state remains in effect, the algorithm scans and rescans the text until all the letters of the complete text have appeared in their intended positions. At this point the text is fully legible according to modern dictionary-endorsed spelling and conventional typesetting.

When overboard opens in its current QuickTime version, the verses are set to 'surface' until the text is complete (in fact it may begin to float or sink before this is the case in exceptional circumstances). Once the text has surfaced, the states of the verses are allowed to change according to a quasi random scheme. For example, verses two and three may be set to float, verse one may begin to sink and verse four may remain, unchanging for a time, in a surfacing state. Or the pattern may be entirely different.

The 'sinking' state is the complement to 'surfacing.' The letters alternate or are changed to blank space. If they are changed to blank space, this persists such that if a verse remains long enough in a sinking state, it eventually disappears.

In the 'floating' state, letters alternate according to a simple table constructed, as I say, on the basis of similarity perceived according to one or more scales of properties that identify letters. The table is a composition of my own that compromises over questions of the various types of perceived similarities in order to arrive at a single alternate for each letter. For English (4), these are the pairings that overboard currently uses:

a e
b p
c s
d b
e a
f t
g q

h r
i j
j i
k c
l t
m w
n r

o u
p q
q p
r n
r n
s c
t l

u o
v y
w v
x z
y v
z x

Note that the table is not a reversable code; there are a couple of many-to-one correspondences and the alternate column is not a complete English alphabet.

Thus, when floating, the text, at each letter position can be in one of two states: the position may contain the natural letter or its (similar) alternate. This renders the text both illegible and legible. I would argue that, except when sinking, the text is always legible. Because the alternates are similar they render patterns of letters that suggest relatively familiar word shapes and often close to their 'natural' originals or to other words and near-words.

For example, the line, "to the brim of the water" expressed entirely in alternate letters would be, "lu lra pnjw ut lra velan", while in the course of an actual performance of overboard (see the screen shot below), when its verse was floating, the line was, for one moment, rendered as, "lo lra briw ut tha vatar."

Moreover, if the scheme of alternates is considered as a type of code, it is one that is easily 'cracked,' even within the duration of a typical viewing or performance.

Verses are set in various states and allowed to persist in those states for varying durations. After a time, the configuration of verse states changes: a sinking verse may begin to surface, a floating verse to sink, a surfacing or surfaced verse to float. Meanwhile, visual and audio correlatives of these transliteral processes are also being performed.

Screen shot from overboard with verse 1 'sinking', verses 2 and 3 in slightly differing stages of 'surfacing', and verse 4 'floating'. The musical cursor is in line 4 of verse 2.

The entire performance of overboard, including its visual and audio correlatives, is algorithmically generated. In all cases, the principle was, as I have said before, to use simple but carefully composed algorithms to produce performed outputs with a large measure of significance and affect. The visual component is made from a fragmented picture of the surface of the sea. Each fragment is associated with a particular letter and the fragment associated with that letter is displayed in a corresponding position on the left hand side of the screen whenever the letter appears in the textual transformation.

There is also a visual clue to the performance of the audio correlative that runs through the visual fragments on the left hand side of the screen. A black-on-white rectangle - its changing shape also reflects a mapping to the alphabet - successively replaces the fragments of the sea, running through the visual 'text' like a cursor. It is, in fact, a musical cursor, following the 'melodic' line of the generated musical correlative for overboard.

This musical correlative was composed in collaboration with Giles Perring. Perring selected and built the sounds, tuning and composing them to the sense of the piece and to overboard's algorithmic drivers. The sound is midi-driven. There are only two distinct sounds: a modulated bell tone and a rolling and roaring sound that is based originally on a drum roll (5). The musical cursor scans and rescans the text, but much more slowly, of course, than that of the scanning which performs the literal alternations. The rolling sound sets an overall tone based on the state of the verse which contains the musical cursor (6). The rolling sounds are played whenever the musical cursor reaches a 'space' in the text. Whenever it does this a portamento'ed arpeggio is invoked, alternatively rising and falling, like breath or wind. If the verse containing the musical cursor is sinking, the rolling sounds are low, if the verse is floating, their pitch is mid-range, if the verse is surfacing they are high, roaring wind-like sounds.

To generate the melodic aspect of overboard's music, the musical cursor scans the text and plays a modulated bell sound based on the following scheme. If the letter displayed is 'natural' (a letter in its proper position based on the underlying text), the sound played is from a higher range of notes and the more frequent the letter in the original text, the higher the pitch of the bell sound (i.e. 'e' generates the highest note and 'x' the lowest). If the letter is an 'alternate' then the sound played is from a lower range of notes and the more frequent the letter in the original text, the lower the pitch of the sound (7).

Apart from certain details of weighting and timing applied to the algorithmic processing, this is a more or less complete description of the operative performance of overboard. The piece can be viewed in web-based versions in either English or German. Currently, the PC-targeted version is not as well set as the Macintosh-targeted version.

I leave it to overboard's readers, some of whom will now have read this brief description, to address the work interpretatively - as literary as well as literal art - without, that is, deferring or bracketing overboard's instantiation as a time-based artefact, one whose signifiers themselves are, arguably, both programmed and time-based.

Bibliography

Bradford, William. Of Plymouth Plantation 1620-1647. Edited by Samuel Eliot Morison. New York: Random House, 1952.

Cayley, John. "Literal Art: Neither Lines nor Pixels but Letters." In First Person: New Media as Story, Performance, and Game, edited by Noah Wardrip-Fruin and Pat Harrigan, 208-17. Cambridge: MIT Press, 2004.

Stefans, Brian Kim. "Stops and Rebels: A Critique of Hypertext." In Fashionable Noise: On Digital Poetics, 61-169. Berkeley: Atelos, 2003.

Footnotes

(1.) For more on my usage and sense of 'literal art' see Cayley 2004.

(2.) Often associated with the work of Brian Eno, as it is in the important source for discussions of ambient poetics - at least in the context of writing in networked and programmable media: Brian Kim Stefans' "Stops and Rebels: a Critique of Hypertext."

(3.) Using 'natural' as in 'natural' language, or as when describing musical notes; I am trying avoid implications of essential 'correctness.'

(4.) In the current German version, the same table is used with the addition of alternates for umlaut characters that occur in the translated text. With input from a native speaker a much better table should be produced rendering greater similarity in the generate patterns of letters when the text 'floats' for German readers.

(5.) These handmade sounds are produced as so-called sound fonts that need to be installed in a reader's system in order for the audio component of overboard to be heard as composed. If the two sound fonts are not installed most midi systems will use a 'default' instrument, most often a piano.

(6.) In fact, in the current version of overboard, the musical cursor drops out when it is inside a verse that is 'floating'. I made this change after giving a performance rendition of the piece. A colleague noted that, in her view, the black and white cursor drew extra attention to the left hand visual correlative and distracted from engagement with the text. I decided to allow the piece to take a break from displaying its musical cursor in the expectation that the reader's eyes would cease, at least for a time, to try and follow it, turning back, instead, to the transliterating text.

(7.) The German version of overboard uses letter frequencies based on the German text for these transcodings.

published on dichtung-digital 2/2004