Interactivity and Affect in Intermedial Art: Theorizing Introverted and Extraverted Intermediality - Érudit
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Document généré le 30 jan. 2022 15:52
Intermédialités
Histoire et théorie des arts, des lettres et des techniques
Intermediality
History and Theory of the Arts, Literature and Technologies
Interactivity and Affect in Intermedial Art: Theorizing
Introverted and Extraverted Intermediality
Jasper Sluijs et Anneke Smelik
Programmer Résumé de l'article
Programming Cet article présente deux oeuvres d’art numérique et intermédial : une
Numéro 13, printemps 2009 architecture sonique et interactive, Son-O-House (2004), et un film d’animation
« ouvert », Elephants Dream (2006), qui constituent le point de départ d’une
URI : https://id.erudit.org/iderudit/044046ar analyse de l’intermédialité. Les auteurs observent deux tendances dans les
DOI : https://doi.org/10.7202/044046ar oeuvres d’art intermédiales : l’une que l’on pourrait qualifier d’introvertie et
l’autre d’extravertie. À partir de la pensée sur la différence de Deleuze et les
idées de l’artiste néerlandais Dick Raaijmakers sur l’esthétique de l’art
Aller au sommaire du numéro
intermédial, les auteurs posent que l’intermédialité introvertie se dirige vers
l’intérieur et attire l’attention réflexive sur les relations intermédiales de
l’oeuvre, tandis que l’intermédialité extravertie se dirige vers l’extérieur et
Éditeur(s) engage l’utilisateur dans un registre affectif. Les concepts d’introversion et
d’extraversion relèvent le dynamisme de l’intermédialité et facilitent une
Centre de recherche sur l'intermédialité
analyse plus spécifique des oeuvres d’art intermédiales.
ISSN
1705-8546 (imprimé)
1920-3136 (numérique)
Découvrir la revue
Citer cet article
Sluijs, J. & Smelik, A. (2009). Interactivity and Affect in Intermedial Art:
Theorizing Introverted and Extraverted Intermediality. Intermédialités /
Intermediality, (13), 177–196. https://doi.org/10.7202/044046ar
Tous droits réservés © Revue Intermédialités, 2010 Ce document est protégé par la loi sur le droit d’auteur. L’utilisation des
services d’Érudit (y compris la reproduction) est assujettie à sa politique
d’utilisation que vous pouvez consulter en ligne.
https://apropos.erudit.org/fr/usagers/politique-dutilisation/
Cet article est diffusé et préservé par Érudit.
Érudit est un consortium interuniversitaire sans but lucratif composé de
l’Université de Montréal, l’Université Laval et l’Université du Québec à
Montréal. Il a pour mission la promotion et la valorisation de la recherche.
https://www.erudit.org/fr/*OUFSBDUJWJUZ BOE "GGFDU
JO *OUFSNFEJBM "SU
5IFPSJ[JOH *OUSPWFSUFE
BOE &YUSBWFSUFE *OUFSNFEJBMJUZ
JA S P E R SLU I JS
ANNE K E SME LIK
* n our study of intermedial art we were struck by the interactive musical-
architectural artwork Son-O-House (Edwin van der Heide and Lars Spuybroek,
2004) and the animated movie Elephants Dream (multiple authors through open-
source technology, 2006). These two digital artworks made us realize that the
concept of intermediality was too broad to understand the specific effects and
affects of contemporary intermedia. How could we account for human movement
shaping both a curvature of arabesques and spatial sound projections? How could
we analyze a collaborative, “open”, movie that we fail to understand, yet that
highly engages us? We therefore took up the challenge to specify and designate
a process that in its very complexity evades specification and designation. This
article thus reflects on the flexibility of intermedial theory.
To better specify intermediality while at the same time doing justice to its
dynamics and complexity, we introduce a focus on the direction of an art work
rather than its meaning. In our view, the direction of an intermedial object can
be “introverted” or “extraverted.” Introverted intermediality is directed inwards,
drawing a reflexive attention to the intermedial relations of the work. Extraverted
intermediality is directed outwards, engaging the subject through an affective
register. Introverted and extraverted intermediality are not mutually exclusive
or each other’s counterpart, but indicate two out of many dynamics of interme-
diality at play in a work of art. We hope that the identification of introverted and
extraverted directions in intermedial objects enables a more specific analysis of
intermediality.
i n t e r m é di a l i t é s s no 13 p r i n t e m p s 2 0 0 9interactivity and affect in intermedial art
In the first section we present our working definition of intermediality. The
second part addresses introverted intermediality in the monumental piece of
interactively sounding architecture Son-O-House, and the third section, extra-
verted intermediality in the animated “open” movie Elephants Dream.
5)& */ #&5 8&&/ 0' */5&3.&%*"-*5 :
Various theorists have stressed the notion of the “in-between” as a crucial charac-
teristic of intermediality. In one of the earliest definitions of intermediality, Dick
Higgins points to the relations between various art forms and disciplines within
the Humanities.1 His writings have been quite influential in German thought
on intermediality in the 1990s.2 For the purpose of our research, we highlight
Jürgen Müller’s focus on the dynamic processes between media that push the
object beyond mere multimediality. Intermediality constructs a conceptual
fusion of media, resulting in a distinctly new dimension and experience that are
not present in the separate media.3 This implies that the actual meaning of inter-
mediality is determined by its context. But context is not sufficient to understand
complex medial interactions. Musicologist Nicholas Cook points out that most
studies of multimedia have focused on similarities between media, rather than
on differences.4 He argues that the connected media relate dynamically to the
1. Dick Higgins, Horizons: The Poetics and Theory of Intermedia, Carbondale-
Edwardsville, Southern Illinois University Press, 1984, p. 15-29.
2. Jürgen E. Müller, Intermedialiät: Formen moderner kultureller Kommunikation,
Münster, Nodus Publikationen, 1996; Jürgen Müller, “Intermediality: a Plea and some
Theses for a New Approach in Media Studies,” in Ulla-Britta Lagerroth, Hans Lund and
Erik Hedling (eds.), Interart Poetics: Essays on the Interrelations of the Arts and Media,
Amsterdam and Atlanta, Rodopi, 1997; Eric Vos, “The Eternal Network: Mail Art, Inter-
media Semiotics, Interarts Studies,” in Lagerroth, Lund and Hedling (eds.), 1997; Werner
Wolf, The Musicalization of Fiction: a Study in the Theory and History of Intermediality,
Amsterdam and Atlanta, Rodopi, 1997; Yvonne Spielmann, Intermedialität: das System
Peter Greenaway, München, Wilhelm Fink Verlag, 1998; Yvonne Spielmann, “Intermedia
in Electronic Images,” Leonardo, vol. 34, n° 1, February 2001, p. 55-61; Peter Wagner (ed.),
Icons, Texts, Iconotexts: Essays on Ekphrasis and Intermediality, Berlin and New York,
Walter de Gruyter Verlag, 1996; Werner Wolf, The Musicalization of Fiction: a Study in the
Theory and History of Intermediality, Amsterdam and Atlanta, Rodopi, 1999; Peter Zima,
Literatur Intermedial: Musik-Malerei-Photographie-Film, Darmstadt, Wissenschaftliche
Buchgesellschaft, 1995.
3. Müller, 1996, p. 83-89.
4. Nicholas Cook, Analyzing Musical Multimedia, Oxford, Oxford University Press,
1998. His work has been sadly neglected in studies on intermediality.interactivity and affect in intermedial art
other media in a signifying practice through internal difference. Intermediality
can then be understood as a contest or struggle between media, reframing one
another through their interaction.5 Cook’s notion of medial signification through
conflict, flux and difference refers to the French philosophers of difference.
In the first issue of Intermédialités, Éric Méchoulan focuses on the ontological
status of the “inter” itself, indicating that the term “intermediality” in fact doubles
up its principle because “inter” and “medium;” both mean an in-between: “[…]
pourquoi ajouter encore ‘inter’, comme ‘l’entre de ce qui est au milieu’?”6 He
argues that media are brought together through different relations to time: an
intermediality that comes after the media as its effect; an intermediality that
precedes the media; and an intermediality that is present, im-mediate, to the
medium. In the same issue, Henk Oosterling argues that the in-between does
not have a locus somewhere between the media, since it cannot be localized.
The meaning of intermedia is thus always already mediated. This implies, in the
first place, that a medium has no conceptual meaning in itself, and secondly, that
intermedial meaning is instantly conveyed through mediation. In Oosterling’s
inimitable phrasing: “immediately mediated mediating immediacy.”7
The key philosophers of the in-between are, of course, Deleuze and Guattari.
For them, the in-between is the place of constant instability where things “pick up
speed.”8 By privileging difference over similarity, dynamic multiplicity over single
linearity, and the rhizome over hierarchy, Deleuze and Guattari offer a toolbox
for understanding intermediality as the in-between that makes a difference.
The dynamics of intermediality can thus be situated in its in-betweeness,
in the very combination of media that produces an altogether new experience.
Crucially, this experience is the result of the “inter,” which can never be reduced
to any of the individual media that prefigures it. As Sylvano Santini remarks,
intermediality is found in “[…] le mystère du mouvement substantiel […];” 9 the
mystery being that movement is perceived and experienced between the media
5. Ibid., p. 103.
6. Éric Méchoulan, “Intermédialités: le temps des illusions perdues,” Intermédialités,
n° 1, “Naître,” Spring 2003, p. 22.
7. Henk Oosterling, “Sens(a)ble Intermediality and Interesse: Towards an Ontology
of the In-between,” Intermédialités, n° 1, “Naître,” Spring 2003, p. 42.
8. Gilles Deleuze and Félix Guattari, A Thousand Plateaus: Capitalism and Schizo-
phrenia, trans. Brian Massumi, London and New York, Continuum, 2004, p. 28.
9. Sylvano Santini, “La perception du mouvement entre disparition et apparition:
réminiscence mallarméenne de l’intermédialité,” Intermédialités, n° 10, “Disparaître,”
Autumn 2008, p. 45.interactivity and affect in intermedial art
rather than being something within the isolated medium itself. Thus, the expe-
rience of movement creates intermediality: “le résultat de la synthèse de deux
médias.”10 The relation between the media is a fundamentally unstable process,
the outcome of which is never certain. This raises the question how meaning is
constructed in an intermedial experience. Or whether it is still appropriate to
speak in terms of “meaning.” Perhaps intermediality is better understood, with
Deleuze, in terms of “what it does” rather than “what it means.”11 To enhance our
understanding of the experience that is produced by intermedial connections, we
need to delve deeper into the in-between of intermediality. Introducing the terms
of “introverted” and “extraverted” intermediality might help us to do so.
As we have seen, the in-between involves instability and movement. We
therefore want to refocus on the direction of intermediality by questioning how
the media relate to one another, and how the intermedial object relates to the
audience. Direction points to a general development on a certain track without
paying attention to a final stage. In defining a possible direction of intermediality,
the continuous development between the media is left intact, while at the same
time a non-representational and non-teleological specification is enabled. We
suggest two possible directions: introverted and extraverted.
“Introverted” intermediality is directed inward, in that it draws attention
to its own principle of intermediality as an in-between of different media. As
such, it is meta-reflexive. Introverted intermedial relations can be formal, abs-
tract and stylized, but they still require the participation of the spectator or user.
“Extraverted” intermediality is directed outwards, in that it draws attention to its
expressive features. Its intermedial relations are diffuse and moving, triggering
affect and sensation for its audience. It thus results in a certain je ne sais quoi that
is highly engaging.
The distinction between introverted and extraverted intermedia is fluid
and flexible. Our aim is not to postulate sub-categories of intermediality, but,
more modestly, to offer some suggestions for possible further classification of
intermediality. In the following two sections we want to experiment with this
classification of intermedia by further theorizing introverted and extraverted
intermediality in relation to digital art works.
10. Ibid., p. 51.
11. Claire Colebrook, Understanding Deleuze, Crows Nest, Allen & Unwin, 2002,
p. xiiv; Simon O’Sullivan, Art Encounters Deleuze and Guattari: Thought Beyond
Representation, Basingstoke and New York, Palgrave Macmillan, 2006, p. 43.interactivity and affect in intermedial art
5)& )064& 8)&3& 406/%4 -*7 &
Son-O-House is a public art work situated in a small park of an industrial complex
reserved for IT and new media companies near Eindhoven in the Netherlands,
the headquarters of multinational Philips. This piece of interactive “sounding”
architecture is made by architect Lars Spuybroek and electronic musician Edwin
van der Heide. Son-O-House features an interplay between architecture and
music through digital technology, testifying to a complex intermedial process
that involves interactivity.
The building is approximately twenty meters long by six meters wide by four
meters tall, placed on a concrete surface that roughly follows the formation of
the object. Son-O-House does not contain a single straight line, including the
slightly sloped surface on which it stands. The curves of the building are covered
with stainless steel plates and wire netting, leaving parts of the structure semi-
transparent while the steel plates cast shadows inside. The object has an entryway
on each side, although it is not possible to enter Son-O-House from any way
standing straight. Once inside the building, the visitor is dazzled by the multi-
tude of curved lines and the various vanishing points guiding the eyes to all sorts
of directions. Forming an amorphous and rhizomatic whole, Son-O-House does
not have a fixed center. Speakers in the building are attached to curves, covered
in balls of wire netting to meet the overall design of the object.
Its architectural construction was rather complex and followed four stages.12
In the first stage, the movements of a person walking through a house were
recorded. In the second stage, a recorded sequence of these movements was
translated methodically into incisions in paper strips. When these strips were
stapled together and placed on a surface, they automatically formed a complex
structure of bent arches that was generated by the cut material itself. In the third
stage, the form erected by the paper strips was met by perpendicular bands. This
stage, too, generated a curvature, which was then digitalized into a computer
model. By computer animation the ends of the curves were knotted together.
The form generated in the computer was then transferred to stainless steel rib
curves.
The work is situated explicitly in-between architecture and music. The
aural aspects of Son-O-House consist of a “complex feedback system in which
visitors are both listeners and interpreters.”13 Composer Van der Heide’s aim
12. Lars Spuybroek, “Son-O-House, a House where Sounds Live,” in Lars Spuybroek
(ed.), NOX: Machining Architecture, London, Thames & Hudson, 2004, p. 174-178.
13. Ibid., p. 196.interactivity and affect in intermedial art
Fig.1 : Son-O-House does not contain a single straight line. Paper model 2004. Courtesy NOX/Lars Spuy-
broek.
in collaborating with Spuybroek was “to get rid of the distinction between
architecture and sound.”14 This called for a spatial interpretation of sound, by
which it could act on the same plane as architecture. In order to make the
sounds in Son-O-House into a spatial composition, they are not based on iden-
tical repetition but rather on internal difference, that is, on constant variation
and development. This process is constituted through interactivity. The Son-
O-House is divided into five sound fields which each have four loudspeakers.
Throughout the building twenty-four sensors are placed at different heights and
in different positions.
The sensors track movements of the visitors and alter the sounds of the
speakers closest to the particular sensors accordingly. The movement around the
sensors is statistically analyzed, to find out which places in the building attract
more movement and which ones do not. The sounds that are played at “popu-
lar” places are then slowly transferred to less popular spots. Spuybroek and Van
der Heide argue that if in these less popular spots the sensors gradually track
14. Edwin Van der Heide, “Son-O-House: Interactive Sounding Architecture,” in
Joke Brouwer, Arjen Mulder and Anne Nigten (eds.), aRt&D: Research and Development
in Art, Rotterdam, V2_publishing/NAi Publishers, 2005, p. 76.interactivity and affect in intermedial art
Fig. 2 : Paper strips form a complex structure of bent arches, generated by the material itself. Picture of
model Son-O-House 2004. Courtesy NOX/Lars Spuybroek.
more movement, this can be understood as the result of sound transition. In such
a case, the system stores the sounds for possible future re-use. A double-sided
interactive process is thus at work: by their movements visitors leave traces in
the continuously changing composition of Son-O-House, while at the same time
the system learns which sounds are most effective in triggering and influencing
human movement. It is important to note that the musical interaction is indirect:
a visitor does not drastically alter the featured sounds, nor will she easily perceive
the influence she has. Instead, the overall sound composition gradually changes
due to human movements. Son-O-House builds on the generative difference
between sound and architecture, allowing both media to meet through spatial
connections. Mark Hansen’s analysis of Son-O-House astutely points out that the
musical and architectural interaction both define space through movement, ininteractivity and affect in intermedial art
different interactive processes that are nevertheless interconnected.15 Here we
may recall Santini’s insight that it is the movement between the media that
results in intermediality, where neither medium is a “[point] de départ ou
d’arrivée; [l’intermédialité] veut somme toute saisir ensemble événement et
inachèvement.”16 In Son-O-House the recorded bodily movement is coupled
to the material self-organization of the paper strips, resulting in a form—the
building—that in itself enables movement within the construction. Moreover,
the movement in the architectural form of the building facilitates the subject’s
interaction with the composition of Son-O-House. Finally, through the subject’s
movement the internal composing system changes, which in turn has the poten-
tial to influence the movement of people in the building. Movement in the
artwork Son-O-House brings about an introverted form of intermediality that
approaches, in Santini’s words, “la fluidité de l’être;” the fluid process of life
itself.17
5)& 3&'-&9*7 *5 : "/% */5&3 "$ 5*7 *5 : 0' */5307 &35&% */5&3.&%*"-*5 :
Son-O-House is obviously an intermedial object, where two media, sound and
architecture, meet without any relation of hierarchy. As composer Edwin van
der Heide remarked, the sounds act either in analogy or counterpoint to the
architectural space, with neither medium defining the significance of the actual
artwork. The question is how Son-O-House can be understood as a form of
introverted intermediality.
As we recall, difference is important in understanding intermediality as the
in-between. In Difference and Repetition, Deleuze develops an alternative to
the traditions of transcendence and representation, by carefully delineating the
process in which difference and repetition coalesce.18 In a dynamic interplay,
an incorporated difference is repeated, generating a series of independent and
positive repetitions, each with a productive and creative potential. Any repeti-
tion of a given in a series is only seemingly an identical repetition. Repetition
15. Mark Hansen, “Embodiment: the Machinic and the Human,” in Brouwer,
Mulder and Nigten (eds.), 2005, p. 155-160.
16. Santini, 2008, p. 52.
17. Ibid., p. 53.
18. Gilles Deleuze, Difference and Repetition [1968], trans. Paul Patton, London
and New York, Continuum, 2004, p. 82.interactivity and affect in intermedial art
therefore operates on two planes: internally, repetition is constituted through
difference, while externally, the resemblance is only a simulacrum of identity.19
The ideas presented in Difference and Repetition once again emphasize
the need to approach intermediality through the potential that arises out of
the difference between media. Deleuze offers a perspective on the internal
mechanisms of intermediality. This allows us to understand Son-O-House as an
artwork in which the different parameters of sound and architecture function
as a creative nucleus. Architect Spuybroek and composer Van der Heide have
constructed a work in which sound and shape are creatively juxtaposed, resulting
in an experience that builds on internal difference. While Spuybroek’s architec-
tural design is fundamentally material—generated by the curvature of the paper
strips—it is both non-linear and non-representational. Son-O-House does not
represent an external intention or model, because its form is constructed by
coincidence. No single shape is identical to another, as all shapes originate in
recorded human movement. Similarly, the composition of Son-O-House’s sound
fields is not determined by repetition or variation on a basic musical theme.
Instead, the composition is dynamic and its constant variations are due to a
mechanism of indirect interaction: repetition through difference, difference
through repetition. The non-representational and interactive nature of both
sound and architecture in this work enables an “open,” non-hierarchical relation
between the media involved.
In our attempt to analyze a complex work like Son-O-House more closely
we turn to the theoretical work by electronic composer Dick Raaijmakers.
Raaijmakers experimented with popular electronic music for the Philips
physics laboratories and developed analog sampling, looping and sequencing
devices as early as the late 1950s. His current work moves between three artistic
disciplines—music, theater and visual art—in a highly unconventional sense
reminiscent of Fluxus.20 Raaijmakers has extensively documented and theorized
his work and artistic principles. His views on the visual aspects of electronic
music are particularly suitable in further classifying introverted intermedia-
lity, allowing us to go beyond the boundaries of intermedial theory. According
19. Ibid., p. 155.
20. See the documented CD releases Popular Electronics: Early Dutch Electronic
Music from Philips Research Laboratories 1956-1963 (2005, Basta Music) and The
Complete Tape Music of Dick Raaijmakers (2006, NEAR/Domenus) for further informa-
tion on Raaijmakers œuvre. See also the English language edition of Dick Raaijmakers’
monograph: Joke Brouwer and Arjen Mulder (eds.), Dick Raaymakers, a Monograph,
Rotterdam, V2_ Publishing, 2008.interactivity and affect in intermedial art
to Raaijmakers, the outcome of a combination of media should meet three
criteria: it should be self-referential; consist of an elementary and simplified
form; and induce active participation by the audience. In our view, these
requirements constitute the kind of introverted intermediality that can be
found in Son-O-House.
Raaijmakers used modernist painter Piet Mondriaan’s views on music,
which propose a non-hierarchical structure of music based on the primary
colors, called “neo-plasticism.” There should be three primary sounds, to be
alternated with three sounds representing the non-colors white, grey and black.
The alteration between colored and non-colored sonic sequences should not be
the result of any hierarchy or linearity, but the distinction should function as an
organizational principle between “tone” and “—tone”. Moreover, this principle
abandons traditional notions of harmony and melody. Mondriaan formulated it
as follows: “One sound will be immediately followed by another—which is truly
‘other’ and which is by no means the ‘rest’ found in old music.”21 The resem-
blance to Deleuze’s views is remarkable, as he formulated a distinction between
being and ¬being, where non-being is by no means a negation of being.22
Raaijmakers rightly remarks that as music will always contain a temporal
element, it will therefore produce some sort of linearity: the horizontal move-
ment through time. When combining sounds with images some problems arise:
while sound may be temporal a priori, images do not have temporal parame-
ters.23 How then to combine these two, without one of the media becoming
more important than the other? Raaijmakers works out an elaborate theory
that features a spatial understanding of sound. He argues that the melody of
music follows a horizontal direction, whereas in harmony melodic elements
are vertically stacked on top of each other. He introduces a third dimension to
sound: depth. The horizontal (melodic) and vertical (harmonic) movement of
sound forms a two-dimensional plane following the x and y axis. This plane,
however, is two-dimensional only when viewed full frontal. Behind each plane
is an acoustic aggregate that constitutes the tone with a spatial depth on a z axis.
Depth here should not be understood in the more conventional sense of reverb,
but rather as timbre. The spatial aggregate of sound allows for a dimension that
is not necessarily affected by time. According to Raaijmakers, here lies a great
21. Dick Raaijmakers, Cahier “M”: a Brief Morphology of Electric Sound, Leuven,
Leuven University Press, 2005, p. 31.
22. Deleuze, 2004, p. 253.
23. Raaijmakers, 2005, p. 35-38.interactivity and affect in intermedial art
potential for music that is still largely virgin territory. Through the z axis, real
interaction with other—non-temporal—media can be acknowledged to surpass
the temporal linearity that is a given in sound. Rather then strictly following
the linearity of time in melody and harmony, the listener of three- dimensional
music is able to participate more actively and tilt her perspective in order to
perceive the three-dimensional quality of a sound. Hierarchical structures
between composer, score, performer and listening subject are thus contested,
because the subject can experience an object more independently.24
The active role that Raaijmakers designates to the listening subject is
similar to the interactivity of Son-O-House. The work of Spuybroek and Van
der Heide needs the visitor’s active participation in order to establish a crea-
tive difference between media. Without the interactive sound production,
Son-O-House’s architecture could not enable any interaction with its visitors.
Conversely, without the building’s structure that people can traverse in all
directions, no interactive composition would be possible. The media in Son-
O-House thus meet one another through the participatory subject. Raaijmakers
further specifies the active role for the subject in stressing the need for a simpli-
fied form. When distilled to its most elementary form, he argues, audiovisual
composition consists of repetitive pulses. The most fundamental artistic form
possible is the single pulse, which, in order to be perceived, requires a temporal
duration by being followed by a non-pulse. By itself the single pulse means
nothing; it needs a second pulse to gain any meaning. Consequently, it is only
in the act of initiating a second pulse that something can become a composition
or a structure.25 The most basic repetition of a single pulse is necessarily a dif-
ferent entity relative to that single pulse—again we find repetition of difference,
difference through repetition.26
24. Ibid., p. 51-56.
25. In an insightful article Briankle Chang has taken this argument one step further
in a slightly different context. Chang, too, acknowledges that the beginning of a series
can never exist in isolation: if a beginning is not followed by anything else, it cannot
possibly be the beginning of anything at all. Only when the initiative is repeated, can it be
perceived as an original in the first place. Chang deftly points out the implications of this
situation: ontologically, “firstness” actually moves to the third place in line. One can only
recognize the original as original, after its first repetition. The first term of a series marks
identity, the second characterizes difference, while the third marks the beginning of the
series and thus the perception of series as such (Briankle G. Chang, “Deleuze, Monet and
Being Repetitive,” Cultural Critique, vol. 41, Winter 1999, p. 188-190).
26. Raaijmakers, 2005, p. 89-110.interactivity and affect in intermedial art
Since Raaijmakers’ concern is audiovisual art, he subsequently pays atten-
tion to the visual arts. He criticizes the top-down credo of representation through
the subject’s role in mediating the outside world. Raaijmakers exemplifies his
views by distinguishing between the movement and stasis of object and subject
during the playback of a recorded sequence. When the object is in motion while
the subject is not, the subject is passively watching a spectacle that is imposed
on him, as is the case in traditional theater and cinema. This situation changes
when subject and object act the other way round. When the subject has the
ability to move around and through a static object, he can take on an active role
and perceive the object (inter)actively.
For the subject to be able to participate and relate to an artwork, it should
be as elemental, simple and straightforward as possible. Raaijmakers refers back
to the minimalism of 20 th century modernism here, yet he goes beyond mere
formalism by highlighting the active role of the subject. He claims that the
elemental form of an artwork allows for active perception, leading the subject
to the inner mechanisms of the artwork. It may come as no surprise that Raaij-
makers rejects any representational value of the artwork, but rather situates its
value in its intermediality. We can find the same value in Son-O-House, where
a new experience emerges beyond strictly musical or architectural terms. In
drawing attention to its own double mechanism of interaction that is intercon-
nected between the planes of sound and architecture, the object becomes both
self-referential and meta-reflexive. It is because of these qualities that we want to
define Son-O-House as an artwork of introverted intermediality.
Deleuze’s notion of complex repetition through internal difference finds
its aesthetic equal in Raaijmakers’ spatial depth in music. Raaijmakers has
investigated the most elemental form of intermedial composition—a non-
identical series of pulses—to conclude that only through the simplicity of
an object can an active role of the subject be made possible. Deleuze and
Raaijmakers also meet in their critique of representational logic. For Deleuze,
difference forms an alternative to the hierarchy of representational perception.
Raaijmakers approaches representation from the viewpoint of the audience:
opposing passive (representational) perception he offers an alternative of (inter)
active artistic experience. The strength of combining the two theoretical frame-
works lies in the possibility of actualizing the implementations of Deleuze’s
philosophy with Raaijmakers’ ideas on multimedial art.
The interplay of Raaijmakers and Deleuze enables us to understand
introverted intermediality with reference to Spuybroek and Van der Heide’s
Son-O-House. The three parameters—elemental form, self-referentiality, activeinteractivity and affect in intermedial art
participation—help to specify the notion of introverted intermediality. The
requirement of elemental form may need some further explanation, because
at first sight the complex structure of tilted arabesques Son-O-House seems
anything but a plain and simple form. However, the actual building with its
diffuse curvature is only a materialization of a highly abstract and simple form
of cuts in paper strips. In our view, this digitalized abstraction defines the basic
and elemental formation of Son-O-House. Its complex system of interactive
inter mediality is elemental in form, which allows visitors to partake in the
composition: whoever experiences Son-O-House inevitably becomes part of the
construction of the artwork itself. Raaijmakers’ theory (as well as his own art)
favors introversion: the subject is supposed to delve deep into the interior of an
artwork. In the case of an intermedial artwork, that interior is experienced in the
unspecified locus in-between media. Son-O-House does not generate represen-
tational meaning, but produces an intermedial experience that is situated in the
in-between. The self-referentiality and the active participation are thus part and
parcel of introverted intermediality.
"/ 01&/ .07 *&
Our second case study is the short movie Elephants Dream, which allows us
to expand on the concept of extraverted intermediality. An animated movie of
approximately ten minutes, Elephants Dream is the result of seven months of
work by an international core team of seven people (“Project Orange”) aided by
a community of open-source volunteers. It is a co-production between Blender
Foundation—an open source software collective—and Montevideo—a media
art institute, both located in Amsterdam. The idea of Elephants Dream was to
produce an animated movie that is totally “open”: while entirely produced by
open-source software, the source code of the movie itself is also freely available
for anyone to alter the movie’s form or content.27
The basic premise behind the project of Elephants Dream was that only open
source software should be used. Necessary programs in open source format had
to be coded by the filmmakers themselves. The emphasis on technical issues in
the early stages dominated some of the creative developments, which resulted in
a highly unconventional and non-linear process of production. For instance, the
27. Our source for the section on Elephants Dream is the website www.orange.blender.
org (last access March 14, 2010), where Elephants Dream can also be downloaded. We have
also used the “making of” documentary by Thomas Kinast and Christian Huter (2006) and
Project Orange’s presentation at the Blender Conference of October 2006.interactivity and affect in intermedial art
Fig. 3 : The two characters Proog and Emo inhabit a giant machine in Elephants Dream. Copyright
2006, Blender Foundation/Netherlands Media Art Institute / www.elephantsdream.org.
Orange team developed a script along the way—there was never a “final” script
that functioned as point of departure. The creators of Elephants Dreams thus
worked on the production of the movie without ever having a clear idea how its
story would end. The character design also continued to be a “work in progress”
throughout the project. The short movie is a clear example of “mutual reme-
diation” between cinema, music and digital media that Jay David Bolter regards
as an important form of intermediality.28 Moreover, as the project progressed and
other people joined the ongoing process through open source communities on
the Internet, the movie kept changing and transforming itself. When sounds,
music and voice recordings were added in the final production stages, the movie
was again drastically altered. The musical interpretation of the script, for exam-
ple, led to significant changes in the creative production of the movie: because
of the unexpected quality of voice recordings, the creative team had to re-model
the characters to make their physical movement and expression match the voice
actors’ intense performance.
The movie’s title, Elephants Dream, refers to the tradition of making up
bedtime stories for children. The movie’s narrative structure is as fragmented
as a typical bedtime story with open endings and dead ends. Elephants Dream
consists of eight scenes that are neither connected logically nor linked within a
28. Jay David Bolter, “Transference and Transparency: Digital Technology and the
Remediation of Cinema,” Intermédialités n° 6, “Remédier,” Autumn 2005, p. 14.interactivity and affect in intermedial art
unifying narrative. The movie features two male characters, the surly old Proog
and naive young Emo, who inhabit a giant machine together with flocks of
mechanical ducks. The look of the short film is a mix of sci-fi and gothic horror.
The movie never explains where the machine comes from, whether it represents
anything, or what Proog and Emo are doing there. The passages between spaces
and scenes appear out of nowhere. Elephants Dream opens when Proog and Emo
are attacked by harpoon-like objects. Although the viewer expects an explanation
for these attacks, it remains unclear why they take place or who is behind them.
The film ends after another attack by large tentacles and a giant hand. As Emo
falls unconscious, the hand and tentacles gradually shrivel. Proog kneels next to
the (lifeless?) body of Emo: “It is there!” he cries in the howling wind, “I’m telling
you Emo, it is! It is.” This is how Elephants Dream ends. This short description
indicates that a basic premise for a character-driven plot is absent from Elephants
Dream. There is no classical story with a beginning, middle and end, nor is there
a rational center designating its meaning and closure.
Online Blender forums are filled with interpretations of Proog and Emo’s
story, while the developers have communicated their views on its meaning as
well. However, its signification is not the point, because the “open” in “open
movie” does not only refer to the open-source form of Elephants Dream, but also
to its content. The open source technology and the collaborative nature of the
project created a disorganized production process that resulted in a continuous
changing of the script, the sound track and the images. Perhaps unintended
initially, the outcome was a fragmentation of the story shattering any linearity
or representational logic. Denying the viewer any underlying signification, the
movie also blocks identification with the characters. Elephants Dream has mul-
tiple entryways into the rhizomatic narrative, offering its own tautological sense.
The movie ends after a dramatic build-up in the final scenes, but still leaves the
viewer clueless as to what has happened or what will happen next. Is Emo dead?
Is the machine Proog’s imagination? Does Emo’s skepticism prevent him from
actually seeing the machine? These questions will not be answered. This undeci-
dability, its very openness, makes up much of the appeal of the movie. Elephants
Dream thus illustrates what Bolter sees as the desired hybridity resulting from the
remediation of cinema through digital technology.29
The lack of a linear narrative, the impossibility of identification, the absence
of a single meaning, and the denial of closure allow, perhaps surprisingly, for the
emergence of a new experience. This experience can be understood in terms of
29. Ibid., p. 16.interactivity and affect in intermedial art
Fig 4 : Proog kneels down to Emo’s body—a dramatic ending without closure in Elephants Dream. Copy-
right 2006, Blender Foundation/Netherlands Media Art Institute / www.elephantsdream.org.
affect. We use affect in its Deleuzean sense as a moment of intensive quality,30
or as a bloc of sensations waiting to be activated by a viewer.31 Affect is an expe-
rience for the spectator that comes prior to meaning. It is according to Colebrook
“a sensible or sensibility not organized into meaning.”32
Elephants Dream is a highly engaging animated film that produces affects
of surprise, disgust, delight and empathy in the spectator. The harpoons that
suddenly fly by, the creepy inside of the telephone receiver, Proog’s dance on the
typewriter buttons, Proog kneeling next to the body of Emo, are all examples of
an affective experience for the viewer. We contend that affect is not only related
to the non-linearity of the fragmented narrative in Elephants Dream, but also to
its intermediality. As demonstrated, small changes in either image, sound (music,
speech) or text (script) dramatically influenced any of the other media, as well as
the overall structure of the animated film. The dynamic processes between the
media in Elephants Dream construct a conceptual fusion, resulting in a distinctly
new experience that could not evolve from its separate media.33 Affect arises from
30. Gilles Deleuze, Cinema 1: the Movement-Image, trans. H. Tomlinson and B.
Habberjam, London, The Athlone Press, 1986.
31. Gilles Deleuze and Félix Guattari, What is Philosophy?, trans. G. Burchell and
H. Tomlinson, London, Verso, 1994.
32. Claire Colebrook, Gilles Deleuze, London, Routledge, 2002, p. 34.
33. Müller, 1996, p. 83-89.interactivity and affect in intermedial art
this non-localized in-between. Contrary to the introverted intermediality of Son-
O-House that produces a reflexive experience for the visitor, the intermediality of
Elephants Dream is directed outwards, producing a new, affective, experience for
the spectator. Therefore, it is for us an example of extraverted intermediality.
5)& */5&/4*5 : "/% "''&$ 5 0' &953 "7 &35&% */5&3.&%*"-*5 :
In this section we want to further flesh out the concept of extraverted inter-
mediality, which, we argue, is first and foremost affective, leading to a sensory
engagement in the viewer. We showed that meaning is not the point of Elephants
Dream. As Colebrook puts it: “Art may well have meanings or messages but what
makes it art is not its content but its affect…”34 In his books on cinema, Deleuze
calls for a productive analysis of film in terms of affect rather than in terms
of representation.35 Cinema, like any art, is never just representation but insti-
gates always a process of affect and transformation. The affect of the rhizomatic
Elephants Dream communicates an artistic sensation, that, in our view, lies in
its aesthetic intermediality. Hence, let us turn to the notion of affect.
In Deleuze and Guattari’s aesthetics a “bloc of sensations” constructs the
artifact. They emphatically state that it is sensation, and only sensation, that
can produce an artifact. Sensations are forces that are immediately and directly
experienced through the body itself, and not by any representational model.
The sensations are the result of a compound of affects and percepts. Percepts
are interpreted as non-representational perceptions, out of which an affect can
arise. An affect is also non-representational—it lies in-between affections, and
dynamically connects subjective states.36 Affects “are no longer feelings or affec-
tions, they go beyond the strength of those who undergo them.”37 Sensations,
as bundles of affects and percepts, therefore contest the linear relation between
subject and object. We can now understand the disjunction in Elephants Dream
between the representational percepts and the non-representational affects that
they trigger. On a representational level, the viewer may wonder about the
nature of the machine Proog and Emo inhabit, why the mise en scène resembles
larger than life switchboards or camera lenses, or how this relates to the overall
34. Colebrook, 2002, emphasis in original, p. 24.
35. Deleuze, 1986; Gilles Deleuze Cinema 2: the Time-Image, trans. H. Tomlinson
and R. Galeta, Minneapolis, University of Minnesota Press, 1989.
36. Gilles Deleuze, Spinoza: Practical Philosophy, trans. Robert Hurley, San Francisco,
City Lights Books, 1988, p. 49.
37. Deleuze and Guattari, 1994, p. 164.interactivity and affect in intermedial art
narrative. These representational musings are, however, irrelevant as they do not
make sense into a coherent narrative. Instead, the fragmented nature foregrounds
the audience’s experience of irrational and non-representational percepts and
affects. Elephants Dream appeals to us because of its loose ends and the shiver
down our spine. Hence, the point here is not meaning, but affect.
Brian Massumi reworks Deleuzean affect through a distinction between
the quality—content, indexing to conventional meaning—and intensity—the
effective strength—of a stimulus.38 Intensity is embodied in purely automatic
and unconscious reactions, through the skin and nerves, while quality requires
a conscious action of the subject to relate to the narrative or aesthetic structure
of an object. Affect is deeply localized in the body, as it goes through the brain
before being measurable in heartbeat, breathing, goose pimples or other physical
reactions. Quoting empirical studies in neurology, Massumi argues that a bodily
event has its origin in the unconscious before resulting in an active and conscious
completion. It is in this unconscious state that he locates the affective sphere of
intensity. An actual expression is the result of a virtual affective reservoir of inten-
sities emerging into actuality. The reservoir of intensity is filled with habits, that
is, with past actions and perceptions in body and brain. These are communicated
as potential stimuli, leading to action. Massumi claims that an affective virtual
plane is the intermediate between potentiality and actuality: potential stimuli
are pressed into the actual sphere. This process takes place through affect. Affect
does not have a linear locus in between the potential and the actual, but resides
as an amorphous passage that is inseparable from the two planes it connects.
The effect of intensity is therefore always rhizomatic, that is, non-linear and non-
structural. As for Deleuze, Massumi’s affect lies in-between stages, in this case
between the potential and the actual.
By focusing on intensity, Massumi further specifies the role of affect within
sensation, which is so important for Deleuze’s and Guattari’s views on art. Art
is for them “the language of sensations” that disengages “the triple organiza-
tion of perceptions, affections and opinions.” An art work is made up of affects,
percepts and blocs of sensation that “take the place of language.” It is only
through sensation that a work of art can “stand up on its own,” meaning that
it can exist independently of representation.39 With the emergence of a bloc of
sensations, an artwork reaches its goal of offering an alternative to the logic of
38. Brian Massumi, “The Autonomy of Affect,” Cultural Critique, n° 31, “The Politics
of Systems and Environments”, Part II, Autumn 1995, p. 83-109.
39. All quotes in this passage are from Deleuze and Guattari, 1994, p. 164-167.interactivity and affect in intermedial art
representation and the linear system of meaning. One of the means of achieving
a bloc of sensations is through intermediality. As we have seen in Elephants
Dream, intermediality implies the rhizomatic emergence of an original expe-
rience from the in-between of different media, effectuated through the absence
of a linear narrative, subjective identification, single meaning, and final closure.
Within the rhizomatic in-between of an intermedial percept, affect plays a
pivotal role. Affect is the potential that unfolds in the actual experience of the
viewer, because she is not so much interested in where Proog and Emo’s super-
sonic elevator ride will take them, but rather fascinated by the elevator slowly
falling apart when it reaches the outside of the machine before slamming down
on the ground. An open-ended affect of surprise and confusion mixed with
horror arises when Emo pushes walls away from him and eventually makes an
entire room break down by massive tentacles that appear out of nowhere. As a
moment of intensive quality, affect provokes a non-linear, dynamic and rhizo-
matic experience of the story. As we have seen, the asynchronous production
of Elephants Dream, in which image, sound and text constantly rework each
other, results in an animated movie stripped of linearity and rationality. In such
circumstances, affect is foreground, while the more laborious process of signi-
fication is relegated to the background. In our view, then, affect in Elephants
Dream depends on its radical intermediality.
We want to argue that the complexity and richness of intermediality produce
affect in the audience. In such cases, we propose to specify intermediality as
extraverted, because its direction is geared towards the outside world. The vital
characteristic of what we have termed extraverted intermediality points to the
notion of affect as an impersonal intensity that establishes a relation to the outside
world, beyond the merely subjective or personal. The affective level of extraverted
intermediality thus requires the spectator to ascertain an experiential relation to
the film or artwork. The regime of extraverted intermediality creates a different
mode of creative spectatorship, pointing to the transformative power of affect.
508"3%4 " 5)&03 : 0' */5307 &35&% "/% &953 "7 &35&% */5&3.&%*"-*5 :
The architectural-musical artwork Son-O-House and the open animated movie
Elephants Dream pose a challenge for research on intermediality, because they
border on the fringes of the media that constitute them, resulting in a medial
interplay that conventional theory can barely grasp. As such, they are examples
of the precarious in-between of intermedial connections. While the in-between
opens these works up to various interpretations, it also enables and perhapsinteractivity and affect in intermedial art
demands new steps in theoretical frameworks. In this article we have undertaken
such steps by specifying a subdivision of intermediality that signals a direction
rather than a meaning: introversion and extraversion.
On the basis of an analysis of Son-O-House we have argued that introverted
intermediality is directed inwards in its self-reflexivity. The artwork invites the
visitor to reflect upon the in-between of the media involved, architecture and
music. Self-reflexivity is not necessarily a passive enterprise; on the contrary, it
is induced by the interactivity of the artwork. In fact, the building and music
together activate the visitor whose movements and actions are the sine qua non
for the intermedial artwork to take effect. On the basis of an analysis of Elephants
Dream we have proposed that extraverted intermediality is directed outwards
in its affectivity. Affect offers an alternative to narrative perception and linear
closure and thus challenges more traditional subject positions for the spectator
or user. Affect as an effect of extraverted intermediality may help to rethink the
spectator or user position in terms of affinity and intensity, rather than distance
and detachment.
The aim of our article has on the one hand been methodological: by
approaching the introverted and extraverted directions of intermedial artworks
we have hoped to analyze their dynamic and rhizomic registers. On the other
hand, our aim was to further develop the concept of intermediality and make
it more precise, productive and effective for analyzing contemporary artworks.
In our view, introversion and extraversion are helpful tools in understanding
intermediality as a theorem. The case studies illustrate how introversion and
extraversion in intermediality can potentially work. Introversion and extraversion
are, however, merely two possible directions of intermediality. There may indeed
be other directions of intermediality that could be distinguished and theorized:
circular, upward, downward, intensive, extensive, rhizomic. Future works of art
will certainly point to a myriad of intermedial directions, inviting visitors or
spectators to new forms of reflexivity and affect, and challenging academics to
create new concepts that acknowledge the dynamics of intermediality.You can also read
