by John
Cayley For some time, my digitally
mediated literary work, my literal art
(1),
has been involved, in terms of one of its most obvious
formalisms, with transliteral morphing from one given text -
transcribed in machine-encoded alphabetic script - to
another. I have developed The algorithms are not
particularly 'complicated,' 'powerful,' or 'intelligent,'
even given the restricted metaphorical tenor of these
apparently strong adjectives when they appear in the
language of programmers and students of technoculture. I am
just enough of a programmer to know the limitations of what
I am doing in these terms. My investment lies in the belief
that relatively simple algorithmic manipulation of basic
low-level, para-semantic linguistic systems is able,
significantly, to yield rhetorical and, indeed, aesthetic
effects which can be correlated with their
programmatological generators. For example, I would claim
that the iterative transliteral morphs between related texts
- texts that might be seen, for example, as rewrites in
differing styles - will reveal abstracted underlying
structures supporting and articulating the 'higher-level'
relationships between the texts. overboard began with
an inflection of my original morphing algorithm and its
effects. I became interested in letter substitution that
would be in some sense 'minimal,' the substitution of one
letter for another that is as similar as possible - similar,
that is, according to any culturally perceivable scale of
characteristics defining a letter's 'identity': its form,
the linguistic sound(s) to which it refers, its position in
any shared literal schema (such as the alphabet itself and
its traditional ordering), and so forth. The particular
transliteral process that developed from this inflection of
my original morphing algorithm also became associated with
the possibility of a less usual performance of poetics, one
that was 'ambient,' in the sense of ambient music
(2).
I wanted to make a piece that was unambiguously literary but
that might perhaps hang on wall-mounted flat screen, like a
kinetic literary painting. The viewer or reader would see a
textual image with a recognizable underlying form, but this
would change constantly by way of its minimal letter
substitutions, ideally such that the changes would be barely
perceptible. The piece would seem not to change and yet
always to be different, whenever it was given any
attention. In its present form, the
constant transitions in the text of overboard are too
obvious and too varied to fulfil that particular vision. Its
primary instantiation as monitor-based work has necessitated
a modulated presentation. A wall-mounted version could
still, of course, be made, slowed down and retuned to
recover that earlier intention, but that will have to wait
for some future project. In the meantime, apart from
subliminal, ambient transliteral change itself,
overboard became concerned with legibility, literal
legibility, I'm tempted to say. The screen-based work
produces, successively a set of three metaphorically
implicated textual states, that relate, in a fairly obvious
way, to the representational sense of the underlying piece
of natural-language writing. Thus, the formal procedures
applied to the literal structure of the underlying text,
mirror its semantic representations. The text itself,
extracted and adapted from the account of an incident during
the Mayflower crossing in Governor Bradford's Of Plymouth
Plantation, tells the story of a man who was swept
overboard and later hauled back on board the ship alive. The
literal text enacts processes that I have come to think of
as floating, drowning and surfacing, in terms of literal
legibility. I will not quote the entire
text here, despite its brevity, since as I say, legibility
is at issue for its appreciation and although the text is
not concealed, I am unsure as to whether a reader benefits
aesthetically in appoaching the actual performance of the
work, if they either know or do not know what the underlying
text spells out: ... in a mighty
storm The underlying text is set
out with line and stanza breaks in the manner of poetic
verse. Each of the four verses in the complete text may,
independently, be in any one of the three states: floating,
drowning or surfacing. As overboard opens, in its
current version, the display space is black and all verses
are placed in the 'surfacing' state. This means that the
underlying text is scanned by the surfacing algorithms and
its constituent letters gradually appear on the 'surface' of
the display space in a quasi random sequence, white on
black, the letters slowly accumulate to constitute the
textual field. The text is set in a monospaced (or
fixed-width) font so that letters in the notional textual
grid have regular fixed positions. On the left-hand side of
the screen a visual correlative of the text and its textual
processes is displayed, with small images - fragments of a
photograph of the sea's surface - corresponding, in position
and identity, with the letters of the text. This visual
correlative will be discussed briefly later. overboard's scheme of
minimally distinct alternate letters also operates during
the surfacing algorithm. During surfacing this means that,
at a particular letter position, if the space is blank then
either the 'natural' letter (the letter in this position in
the underlying text) or its alternate may appear
(3).
However, when surfacing, if the natural letter appears it
will persist, whereas its alternate letter will eventually
change to the target letter over time. So long as the
surfacing state remains in effect, the algorithm scans and
rescans the text until all the letters of the complete text
have appeared in their intended positions. At this point the
text is fully legible according to modern
dictionary-endorsed spelling and conventional
typesetting. When overboard opens
in its current QuickTime version, the verses are set to
'surface' until the text is complete (in fact it may begin
to float or sink before this is the case in exceptional
circumstances). Once the text has surfaced, the states of
the verses are allowed to change according to a quasi random
scheme. For example, verses two and three may be set to
float, verse one may begin to sink and verse four may
remain, unchanging for a time, in a surfacing state. Or the
pattern may be entirely different. The 'sinking' state is the
complement to 'surfacing.' The letters alternate or are
changed to blank space. If they are changed to blank space,
this persists such that if a verse remains long enough in a
sinking state, it eventually disappears. In the 'floating' state,
letters alternate according to a simple table constructed,
as I say, on the basis of similarity perceived according to
one or more scales of properties that identify letters. The
table is a composition of my own that compromises over
questions of the various types of perceived similarities in
order to arrive at a single alternate for each letter. For
English (4),
these are the pairings that overboard currently
uses: a e h r o u u o Note that the table is not a
reversable code; there are a couple of many-to-one
correspondences and the alternate column is not a complete
English alphabet. Thus, when floating, the
text, at each letter position can be in one of two states:
the position may contain the natural letter or its (similar)
alternate. This renders the text both illegible and legible.
I would argue that, except when sinking, the text is always
legible. Because the alternates are similar they render
patterns of letters that suggest relatively familiar word
shapes and often close to their 'natural' originals or to
other words and near-words. For example, the line, "to
the brim of the water" expressed entirely in alternate
letters would be, "lu lra pnjw ut lra velan", while in the
course of an actual performance of overboard (see the
screen shot below), when its verse was floating, the line
was, for one moment, rendered as, "lo lra briw ut tha
vatar." Moreover, if the scheme of
alternates is considered as a type of code, it is one that
is easily 'cracked,' even within the duration of a typical
viewing or performance. Verses are set in various
states and allowed to persist in those states for varying
durations. After a time, the configuration of verse states
changes: a sinking verse may begin to surface, a floating
verse to sink, a surfacing or surfaced verse to float.
Meanwhile, visual and audio correlatives of these
transliteral processes are also being performed. 4. The entire performance of
overboard, including its visual and audio
correlatives, is algorithmically generated. In all cases,
the principle was, as I have said before, to use simple but
carefully composed algorithms to produce performed outputs
with a large measure of significance and affect. The visual
component is made from a fragmented picture of the surface
of the sea. Each fragment is associated with a particular
letter and the fragment associated with that letter is
displayed in a corresponding position on the left hand side
of the screen whenever the letter appears in the textual
transformation. There is also a visual clue
to the performance of the audio correlative that runs
through the visual fragments on the left hand side of the
screen. A black-on-white rectangle - its changing shape also
reflects a mapping to the alphabet - successively replaces
the fragments of the sea, running through the visual 'text'
like a cursor. It is, in fact, a musical cursor, following
the 'melodic' line of the generated musical correlative for
overboard. This musical correlative was
composed in collaboration with To generate the melodic
aspect of overboard's music, the musical cursor scans
the text and plays a modulated bell sound based on the
following scheme. If the letter displayed is 'natural' (a
letter in its proper position based on the underlying text),
the sound played is from a higher range of notes and the
more frequent the letter in the original text, the higher
the pitch of the bell sound (i.e. 'e' generates the highest
note and 'x' the lowest). If the letter is an 'alternate'
then the sound played is from a lower range of notes and the
more frequent the letter in the original text, the lower the
pitch of the sound (7). Apart from certain details
of weighting and timing applied to the algorithmic
processing, this is a more or less complete description of
the operative performance of overboard. The piece can
be viewed in I leave it to
overboard's readers, some of whom will now have read
this brief description, to address the work interpretatively
- as literary as well as literal art - without, that is,
deferring or bracketing overboard's instantiation as
a time-based artefact, one whose signifiers themselves are,
arguably, both programmed and time-based. Bibliography
Bradford, William. Of
Plymouth Plantation 1620-1647. Edited by Samuel Eliot
Morison. New York: Random House, 1952. Cayley, John. "Literal Art:
Neither Lines nor Pixels but Letters." In First Person:
New Media as Story, Performance, and Game, edited by
Noah Wardrip-Fruin and Pat Harrigan, 208-17. Cambridge: MIT
Press, 2004. Stefans, Brian Kim. "Stops
and Rebels: A Critique of Hypertext." In Fashionable
Noise: On Digital Poetics, 61-169. Berkeley: Atelos,
2003. Footnotes (1.)
For more on my usage and sense of 'literal art' see Cayley
2004. (2.)
Often associated with the work of Brian Eno, as it is in the
important source for discussions of ambient poetics - at
least in the context of writing in networked and
programmable media: Brian Kim Stefans' "Stops and Rebels: a
Critique of Hypertext." (3.) Using
'natural' as in 'natural' language, or as when describing
musical notes; I am trying avoid implications of essential
'correctness.' (4.)
In the current German version, the same table is used with
the addition of alternates for umlaut characters that occur
in the translated text. With input from a native speaker a
much better table should be produced rendering greater
similarity in the generate patterns of letters when the text
'floats' for German readers. (5.) These
handmade sounds are produced as so-called sound fonts that
need to be installed in a reader's system in order for the
audio component of overboard to be heard as composed.
If the two sound fonts are not installed most midi systems
will use a 'default' instrument, most often a
piano. (6.) In
fact, in the current version of overboard, the
musical cursor drops out when it is inside a verse that is
'floating'. I made this change after giving a performance
rendition of the piece. A colleague noted that, in her view,
the black and white cursor drew extra attention to the left
hand visual correlative and distracted from engagement with
the text. I decided to allow the piece to take a break from
displaying its musical cursor in the expectation that the
reader's eyes would cease, at least for a time, to try and
follow it, turning back, instead, to the transliterating
text. (7.) The
German version of overboard uses letter frequencies
based on the German text for these transcodings.
O V E
R B O A R D
An Example of
Ambient Time-Based Poetics in digital art
overboard
by John Cayley, with Giles Perring, is an example
of literal art in digital media that demonstrates
an 'ambient' time-based poetics. There is a stable
text underlying its continuously changing display
and this text may occasionally rise to the surface
of normal legibility in its entirety. However,
overboard is installed as a dynamic
linguistic 'wall-hanging,' an ever-moving 'language
painting.' As time passes, the text drifts
continually in and out of familiar legibility -
sinking, rising, and sometimes in part, 'going
under' or drowning, then rising to the surface once
again. It does this by running a program of simple
but carefully designed algorithms which allow
letters to be replaced by other letters that are in
some way similar to the those of the original text.
Word shapes, for example, are largely preserved. In
fact, except when 'drowning,' the text is always
legible to a reader who is prepared to take time
and recover its principles. A willing reader is
able to preserve or 'save' the text's
legibility.
various
algorithms, based on
literal identities and their sequenced transposition, to
perform these morphs in software-driven systems
(www.shadoof.net/in). My intention has been, in part, to
interrogate certain relationships between the granular or
atomic structures of alphabetically transcribed language and
the critically or interpretatively discoverable rhetorical
and aesthetic effects of literature.
a man came above board
and was thrown into the sea
but he caught hold of the halyards
which hung over board
and held his hold
though he was many fathoms under water
till he was hauled up
to the brim of the water ...
b p
c s
d b
e a
f t
g q
i j
j i
k c
l t
m w
n r
p q
q p
r n
r n
s c
t l
v y
w v
x z
y v
z x
Screen shot from overboard with verse 1 'sinking',
verses 2 and 3 in slightly differing stages of 'surfacing',
and verse 4 'floating'. The musical cursor is in line 4 of
verse 2.
Giles
Perring. Perring
selected and built the sounds, tuning and composing them to
the sense of the piece and to overboard's algorithmic
drivers. The sound is midi-driven. There are only two
distinct sounds: a modulated bell tone and a rolling and
roaring sound that is based originally on a drum roll
(5).
The musical cursor scans and rescans the text, but much more
slowly, of course, than that of the scanning which performs
the literal alternations. The rolling sound sets an overall
tone based on the state of the verse which contains the
musical cursor (6).
The rolling sounds are played whenever the musical cursor
reaches a 'space' in the text. Whenever it does this a
portamento'ed arpeggio is invoked, alternatively rising and
falling, like breath or wind. If the verse containing the
musical cursor is sinking, the rolling sounds are low, if
the verse is floating, their pitch is mid-range, if the
verse is surfacing they are high, roaring wind-like
sounds.
web-based
versions in either
English or German. Currently, the PC-targeted version is not
as well set as the Macintosh-targeted version.
published
on dichtung-digital 2/2004