Temporal ventriloquism: crossmodal interaction on the time 2. Evidence from sensorimotor synchronization
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
International Journal of Psychophysiology 50 (2003) 157–163
Temporal ventriloquism: crossmodal interaction on the time
dimension
2. Evidence from sensorimotor synchronization
Gisa Ascherslebena,*, Paul Bertelsonb,c
a
Max Planck Institute for Psychological Research, Infant Cognition and Action, Amalienstr. 33, Munich, D-80799, Germany
b
Laboratoire de Psychologie Experimentale, Universite libre de Bruxelles, Belgium
c
Donders Laboratory for Cognitive Neuroscience, Tilburg University, Netherlands
Received 21 November 2002; accepted 10 April 2003
Abstract
In two experiments, we measured audio-visual crossmodal attraction on the time dimension, using a sensorimotor
synchronization task. Synchronization performance made it possible to split up the total crossmodal attraction
(demonstrated in earlier studies through inter-modal temporal order judgments) into its modality-specific components,
the auditory bias of the visual event’s perceived time of occurrence and the visual bias of the auditory event’s
perceived time of occurrence. Participants were asked to produce tapping movements in synchrony with a sequence
of isochronously repeated pacing signals. In Experiment 1, pacing signals were light flashes, each preceded or
followed, at one of several stimulus onset asynchronies (SOAs), by an auditory distracter that the participant was
instructed to ignore. The timing of the tap was, in spite of that instruction, strongly biased toward the distracter. In
Experiment 2, the converse task was used. The pacing signals were auditory and the to be ignored distracters, light
flashes. The timing of the taps was biased significantly here also toward the distracter, but to a much lesser extent.
Taken together, these results clearly demonstrate that audition plays a bigger role than vision in temporal ventriloquism
and is probably generally superior to vision for processing the temporal dimension of events.
䊚 2003 Elsevier Science B.V. All rights reserved.
Keywords: Temporal ventriloquism; Sensorimotor synchronization; Time perception
1. Introduction visual events presented in separate locations appear
closer together when the presentations are synchro-
In the classical ventriloquist effect, auditory and nized. The possibility of the reverse phenomenon,
attraction on the time dimension conditional on
spatial proximity was examined in an earlier study
*Corresponding author. Tel.: q49-89-38602-251; fax: q49-
89-38602-199.
of ours (Bertelson and Aschersleben, in press). It
E-mail address: aschersleben@psy.mpg.de was proposed to call that phenomenon ‘temporal
(G. Aschersleben). ventriloquism’ (see also Morein-Zamir et al.,
0167-8760/03/$ - see front matter 䊚 2003 Elsevier Science B.V. All rights reserved.
doi:10.1016/S0167-8760Ž03.00131-4158 G. Aschersleben, P. Bertelson / International Journal of Psychophysiology 50 (2003) 157–163
2002).1 Participants judged the order of occurrence (Bertelson and Radeau, 1981, Exp. 1; Radeau and
of, respectively, sound bursts and light flashes Bertelson, 1987), and sometimes not (Bertelson
delivered at several stimulus onset asynchronies and Radeau, 1981, Exp. 2; Radeau, 1985). Thus,
(SOAs), either both in one single location or in vision appeared to play a much greater role than
two different locations. Temporal discrimination audition in the resolution of auditory–visual spatial
was found better in the latter condition, showing conflict.
that the perceived temporal separation between The situation might be different for temporal
stimuli in the two modalities was effectively ventriloquism. Some suggestion that timing, as
enhanced by spatial separation. A somewhat com- opposed to location, may be processed more effi-
parable pattern has been reported by Spence et al. ciently by audition than by vision has been brought
(2001) for touch and vision as stimulus modalities: by work with the so-called auditory driving effect,
discrimination difficulty, measured by just notice- in which the apparent repetition rate of visual
able differences (JNDs), was also better here when flashes was modified by the simultaneous presen-
one of the two stimuli was applied in either tation of faster or slower series of sound bursts
hemispace rather than both in the same hemispace. (Gebhard and Mowbray, 1959; Welch et al., 1986).
These studies thus converged in demonstrating The converse effect, visual driving of auditory
attractions on the time dimension between stimuli rate, was typically absent or much smaller. In a
in different modalities that were contingent on more recent contribution to the issue, Fendrich and
spatial proximity. However, because they were Corballis (2001) made the participants judge when
based on judgements of relative order of occur- a flash or a click occurred by reporting the clock
rence of stimuli in the two modalities, they provid- position of a rotating visual marker (a task bor-
ed no information concerning the respective rowed from Libet et al., 1983). In one experiment,
contributions of the modalities in bringing about the target, a flash, was accompanied synchronously
the observed temporal attractions. The objective or at a positive or negative SOA by a distracter
of the present study was to achieve separate click. In comparison with the simultaneous click
measurements of these contributions in the case of trials, the flash was seen earlier when preceded by
auditory–visual combinations. the click and later when followed by it. A nearly
In studies of spatial ventriloquism, both the identical pattern was obtained in a second experi-
influence of discordant visual distractors on the ment in which the click was the target and the
localization of auditory targets and the opposite flash the distracter. The effect was only slightly
influence of auditory distractors on the localization smaller than in the first experiment and the differ-
of visual targets, have been assessed in separate ence attained significance only after data for one
experimental conditions (for a review, see Bertel- of the two largest SOAs were excluded. These
son, 1999). A robust visual bias of apparent results thus confirmed that the visual dominance
location was obtained across a range of conditions, generally observed for localization performance
but when it was also measured, the auditory bias does not extend to situations in which timing is
of visual apparent location was found much small- the target dimension, but they fell short of dem-
er. As a consequence, it was sometimes significant onstrating a total shift to auditory dominance.
In the present study, the roles in temporal
1
ventriloquism of two possible components, respec-
These authors used a TOJ task consisting of judging the
order of occurrence of two light flashes presented either in tively, the bias of a visual target’s apparent time
synchrony with two sounds or the first preceded and the of occurrence by an auditory distracter and the
second followed by a sound at one of several SOAs (range bias of an auditory target’s apparent time of
75–225 ms). Temporal discrimination was significantly better occurrence by a visual distracter, were analyzed,
(smaller just noticeable differences) with asynchronous sounds using the synchronization paradigm. In a typical
than with synchronous ones, reflecting presumably some
attraction of flashes’ perceived time of occurrence toward the sensorimotor synchronization task, participants are
sounds. Unfortunately, a further experiment showed that such instructed to tap on a key with, for example, their
attraction occurred only for the second flash. right index finger, in synchrony with a sequenceG. Aschersleben, P. Bertelson / International Journal of Psychophysiology 50 (2003) 157–163 159
of pacing signals following each other at constant (i.e. a reduced negative asynchrony) as compared
time intervals. Performance on such tasks has been to conditions in which the two pacing signals are
studied since Stevens (1886). Generally, taps were presented simultaneously. The effect of an auditory
found to precede the pacing signal by a negative distracter on synchronizing with a visual pacing
asynchrony (or synchronization error) of 20–80 signal was examined in Experiment 1 and that of
ms (Aschersleben and Prinz, 1995). This effect a visual distracter on synchronizing with an audi-
had been described already more than 100 years tory pacing signal in Experiment 2.
ago (e.g. Dunlap, 1910; Johnson, 1898; Miyake,
1902) and has more recently been replicated in a 2. Experiment 1
number of studies (e.g. Aschersleben and Prinz,
1995, 1997; Fraisse, 1980; Kolers and Brewster, In this first experiment, participants had to
1985; Mates et al., 1994; Vos et al., 1995; for an synchronize finger taps with a visual pacing signal
overview see Aschersleben, 2002). The size of the while ignoring an auditory distracter presented
negative asynchrony is dependent on a number of simultaneously or at some SOA before or after the
factors, for example, on the amount of sensory pacing visual signal. If the apparent time of occur-
feedback about the tap available, on the sensory rence of the pacing signal was biased toward the
modality in which this feedback is presented and auditory distracter, this had to be reflected in an
on the modality of the pacing signal. Moreover, influence of the SOA on the synchronization error.
the (relative) timing of the tap can be used as an When the auditory distracter preceded the visual
indicator for the perceived timing of the pacing pacing signal (negative SOA values) an increase
signal. By using the synchronization task, Aschers- in the synchronization error was expected in com-
¨
leben and Musseler (1999) could show, for exam- parison with the zero SOA condition, whereas a
ple, that the perceived timing of a moving stimulus delayed timing of the tap was expected with
is delayed as compared to a stationary flash. If the positive SOA values (auditory signal following
moving stimulus was presented as pacing signal a the visual one).
smaller synchronization error (indicating a delayed
timing of the tap) was observed in comparison 2.1. Method
with conditions in which a brief flash served as
pacing signal. 2.1.1. Participants
Following this general logic, we presented pairs A group of 10 healthy participants (seven
of visual and auditory stimuli with different stim- females and three males, aged 22–34 years; mean
ulus onset asynchronies (SOAs) as pacing signal age 25.1 years; all right-handed) performed the
in a sensorimotor synchronization task to study experiment. The results of two additional partici-
the respective contributions of the visual and the pants had to be discarded from the final analysis
auditory modality in bringing about the temporal (see Results section).
ventriloquism effect. In different tasks, participants
were instructed either to synchronize with the 2.1.2. Experimental situation
visual signal and ignore the auditory signal or to The participant sat with the head in a forehead
do the opposite. The advantage of the synchroni- and chin rest, in front of a black wooden box
zation task is that the timing of the tap can be containing a loudspeaker and an LED lamp. The
used as point of reference for the (relative) timing LED lamp was positioned at about eye level in
of the pacing signal. If, for example, the participant medial location. The loudspeaker was located right
is instructed to synchronize with the visual signal below the LED (vertical distance LED-center of
and the distracting auditory signal is presented the loudspeaker: 8 cm). On each trial, a sequence
after the visual one, then, an attraction towards the of 34 flashes, each 10 ms long, and separated by
auditory signal is expected which should result in 800 ms intervals, was presented. Each flash was
a delayed timing of the visual signal. This delayed preceded or followed by an auditory signal (10
timing should lead to a delayed motor response ms, 400 Hz, sinusoidal waveform). Intensity of160 G. Aschersleben, P. Bertelson / International Journal of Psychophysiology 50 (2003) 157–163
the tones, measured at the distance of participant’s
head, was 67 dBwAx, and luminance of the flashes
was 45 cdym2. The SOA between stimuli in each
pair was constant within a block of trials, but
varied between blocks. The task was to tap in
synchrony with the visual pacing signal with the
right index finger on a silent electrical contact
switch mounted on a wooden board. The switch
consisted in a sensory key, i.e. a metal plate, which
registered the touch of the finger via changes in
the electric field. The response time of the switch
was below 0.5 ms.
Nothing was said to the participants about the
SOA between the visual and the auditory signal.
To insure that they looked at the LED, the color
of the flash changed from white to light green Fig. 1. Experiment 1 (Synchronizing with visual pacing signals
two, three or four times in each trial (at random while ignoring temporally discordant auditory distracters):
Mean asynchronies between tap onset and flash onset in the
positions). Participants had to type on the keyboard different SOA conditions, with brackets indicating between-
the number of these color changes at the end of participants S.E.
the trial. A Hewlett-Packard Vectra QSy20 person-
al computer controlled the presentation of the asynchronies between tap onsets and flash onsets
stimuli via a DyA converter and a Sony TA-F170 were computed for each trial (synchronization
amplifier and also registered the responses (with a error). Negative values indicate that taps preceded
resolution of 1 ms). flashes. To eliminate transfer effects between SOA
conditions, the first trial of each block was never
2.1.3. Procedure included in the analyses.
Seven different SOA conditions were tested in
the experiment (0 ms, "15 ms, "30 ms, "45 2.2. Results and discussion
ms). These conditions were presented blockwise,
each block consisting of seven trials. All partici- The mean values per trial were entered into a
pants started with the 0 ms-SOA condition. The repeated-measures analysis of variance (ANOVA)
order of the remaining six conditions was counter- that distinguished two within-participant factors:
balanced across participants. Instructions required SOA (d.f.s6) and trial number (d.f.s5). SOA
them to start tapping as soon as they picked up was the only factor that reached significance
the beat, and then tap along as precisely as wF(6,54)s33.87, P-0.001x. There was no signifi-
possible. cant difference between the six trials applied for
each condition and no significant interaction wP-
2.1.4. Data analysis values)0.50x. As expected, a dependency of the
The results of two participants were discarded synchronization error on the SOA between the two
from the final analysis because they made more conflicting signals was observed. Negative SOA
than 10% errors in the monitoring task (counting values resulted in an increase in the synchroniza-
color changes), leaving us with the data from 10 tion error whereas under conditions with positive
participants. Data analysis started with the fifth SOAs a delayed timing of the tap was observed
signal in each trial. The initial taps were not as compared to the simultaneous presentation of
included because they were required for the partic- the two pacing signals (SOAs0 ms). Total aver-
ipant to pick up the beat. Hence, the means age asynchronies are depicted in Fig. 1.
reported hereafter refer to the taps accompanying A significant positive linear trend in the ascend-
the remaining 30 signals in each trial. The mean ing sorted levels of the SOA was also foundG. Aschersleben, P. Bertelson / International Journal of Psychophysiology 50 (2003) 157–163 161
3.1. Method
3.1.1. Participants
A new group of 10 healthy participants (five
females and five males, aged 20–30 years; mean
age 25.1 years; all right-handed) performed the
experiment. The results of two additional partici-
pants had to be discarded from the final analysis
(see Results section).
3.1.2. Experimental situation and procedure
The experimental situation and the procedure
were identical to the one applied in Experiment 1,
with the exception that here the participants were
instructed to tap in synchrony with an auditory
Fig. 2. Experiment 2 (Synchronizing with auditory pacing sig- pacing signal, while visual distracters were pre-
nals while ignoring temporally discordant visual distracters): sented at several SOAs.
Mean asynchronies between tap onset and tone onset in the
different SOA conditions, with brackets indicating between-
participants S.E. 3.1.3. Data analysis
Again, the results of two participants had to be
eliminated from the final analysis because they
committed more than 10% errors in the control
wF(l,9)s161.56, P-0.001; regression line ys task (counting the color change of the flashes),
0.93xy47.73; R 2s0.99x indicating a linear leaving us with the data from 10 participants. The
dependency of the synchronization error on the means of the asynchronies between tap onsets and
SOA between the auditory and the visual signal. tone onsets were computed for each trial. Negative
The slope of the regression function (0.93) indi- values indicate that taps preceded tones. To elimi-
cated a huge (93%) influence of the SOA. nate possible transfer effects between SOA condi-
Thus, when the participants were trying to syn- tions, the first trial in each block was not included
chronize with a visual pacing signal, their taps in further analyses.
were strongly biased toward the temporally discor-
dant auditory distracter. This finding contrasts 3.2. Results and discussion
sharply with the very small biasing effects that
auditory distracters have on the location of visual The mean values per trial were entered into a
targets in spatial ventriloquism situations. More- repeated-measures analysis of variance (ANOVA)
over, at more than 93% of the experimental SOA, that distinguished two within-subject factors: SOA
this temporal bias is also larger than the usual (d.f.s6) and trial number (d.f.s5). Only the
visual biases of auditory location, which rarely factor SOA reached significance wF(6,54)s5.76,
exceed 50% of the imposed spatial discordance. P-0.001x. There was no significant difference
between the six trials applied for each condition
wP)0.50x and no significant interaction wP)0.20x.
3. Experiment 2
Again, a dependency of the synchronization error
on the SOA between the two conflicting signals
In this second experiment, participants were was observed. Negative SOA values (with the
assigned the reverse task of the one in Experiment conflicting visual signal preceding the auditory
1. They were asked to synchronize finger taps pacing signal) resulted in a significant increase in
with the auditory signal and to ignore the visual the synchronization error whereas under conditions
signal. with positive SOAs a delayed timing of the tap is162 G. Aschersleben, P. Bertelson / International Journal of Psychophysiology 50 (2003) 157–163
observed as compared to the simultaneous pres- analysis, one needed a common baseline against
entation of the two pacing signals. Total average which to measure each component. In Fendrich
asynchronies are depicted in Fig. 2. and Corballis’ (2001) study with the Libet para-
A significant positive linear trend for the factor digm, the rotating time marker provided that base-
SOA was found wF(1,9)s13.27, Ps0.005; regres- line, which, however, had to be judged by the
sion line ys0.19xy37.46; R 2s0.90x. The results participant. In the present study, the participant’s
indicate again a linear dependency of the synchro- finger taps, which could be measured directly,
nization error on the SOA between the auditory played a similar role.
and the visual signal. However, the slope of the Our synchronization experiments have provided
regression function (0.19) was much smaller than the required measurements of the two possible
the one (0.93) obtained in Experiment 1. temporal biases. In Experiment 1, when partici-
In this second experiment, the participants had pants had to synchronize finger taps with a visual
to synchronize taps with an auditory pacing signal, pacing signal while ignoring a temporally discor-
and temporally discordant visual distracters pro- dant auditory distracter, the timing of the tap was
duced only rather weak biasing effects, much strongly influenced by the SOA at which the
smaller than the auditory biases observed in Experi- conflicting auditory distracter was delivered. This
ment 1. The fact that these small visual biases auditory bias of visual occurrence point amounted
nevertheless reach statistical significance is impor- to more than 90% of the imposed asynchrony. In
tant, for it means that auditory dominance for Experiment 2, when the pacing signal was auditory
temporal processing, even if strong, is not total. and the distracter visual, a much smaller (though
significant) visual bias of the auditory occurrence
4. General discussion point, amounting to less than 20% of the imposed
asynchrony, was obtained.
The phenomenon of temporal ventriloquism that This description in terms of modality-specific
was observed in our earlier study (Bertelson and components provides an interesting extension of
Aschersleben, in press) has important implications the already mentioned mirror-image situation. In
regarding the mechanism of spatial ventriloquism, spatial ventriloquism, the general finding is a
and more generally of other cases of immediate strong visual bias of the apparent auditory location,
crossmodal interactions. Being a sort of mirror and the occasional observation of a much smaller
image of spatial ventriloquism, it suggested that auditory bias of the apparent visual location. In
the roles played by, respectively, spatial distance temporal ventriloquism, we seem to have an
and timing in spatial ventriloquism were to some extremely strong auditory bias of apparent visual
extent inter-changeable. In spatial ventriloquism, occurrence time and a smaller visual bias of
sounds and visual flashes were experienced as apparent auditory occurrence point.
occurring closer together when presented synchro- As mentioned in the Introduction, Fendrich and
nously. In temporal ventriloquism, the apparent Corballis (2001) already obtained a significant
asynchrony becomes smaller when the sources are auditory bias of apparent visual occurrence time,
located closer together. However, a limit on that but one whose size was comparable to that of the
symmetry was created by the different ways in opposite visual bias. This result brought thus no
which the two effects were described. Research on clear support for the auditory dominance for timing
spatial ventriloquism had resulted in measurements notion. One might speculate that the marker locat-
for two separate effects, the visual bias of auditory ing response that was used gave the task a spatial
apparent location and the auditory bias of visual dimension, which would be responsible for the
apparent location, but no similar analysis into more substantial visual temporal bias that was
modality-specific components existed for temporal obtained.
ventriloquism, which was only described as a It will have been noted that our results bring
relative attraction on the time dimension between strong support for the notion of a dominance of
sounds and visual stimuli. To obtain that sort of audition over vision for processing the temporalG. Aschersleben, P. Bertelson / International Journal of Psychophysiology 50 (2003) 157–163 163
aspects of events. While this conclusion needs no Dunlap, K., 1910. Reactions on rhythmic stimuli, with attempt
further elaboration, it must be insisted on that the to synchronize. Psychol. Rev. 17, 399–416.
Fendrich, R., Corballis, P.M., 2001. The temporal cross-capture
dominance in question is relative, rather than of audition and vision. Percept. Psychophys. 63, 719–725.
absolute, given the simultaneous existence of the Fraisse, P., 1980. Les synchronisations sensori-motrices aux
smaller visual bias. The excesses committed earlier rythmes wSensorimotor synchronization to rhythmsx. In:
in name of the then prevalent belief in a general Requin, J. (Ed.), Anticipation et comportement. Centre
visual dominance (see Bertelson, 1999) should not National de la Recherche Scienifque, Paris, pp. 233–257.
Gebhard, J.W., Mowbray, G.H., 1959. On discriminating the
be repeated now about alternative cases of
rate of visual flicker and auditory flutter. Am. J. Psychol.
dominance. 72, 521–529.
Johnson, W.S., 1898. Researches in practice and habit. Stud.
Acknowledgments Yale Psychol. Lab. 6, 51–105.
Kolers, P.A., Brewster, J.M., 1985. Rhythms and responses. J.
We wish to thank Frank Miedreich for program- Exp. Psychol. Hum. 11, 150–167.
Libet, B., Gleason, C.A., Wright, E.W., Pearl, D.K., 1983.
ming and Renate Tschakert for her support in data
Time of conscious intention to act in relation to onset of
collection. cerebral activity (readiness-potential): the unconscious ini-
tiation of a freely voluntary act. Brain 106, 623–642.
References ¨
Mates, J., Muller, ¨
U., Radil, T., Poppel, E., 1994. Temporal
integration in sensorimotor synchronization. J. Cognit. Neu-
Aschersleben, G., 2002. Temporal control of movements in rosci. 6, 332–340.
sensorimotor synchronization. Brain Cognit. 48, 66–79. Morein-Zamir, S., Soto-Faraco, S., Kingstone, A.F., 2002.
Aschersleben, G., Prinz, W., 1995. Synchronizing actions with Deconstructing temporal ventriloquism: How do sounds
events: the role of sensory information. Percept. Psychophys. capture the perceived occurrence of lights. International
57, 305–317. Multisensory Research Forum, 3rd Annual Meeting, Geneva,
Aschersleben, G., Prinz, W., 1997. Delayed auditory feedback Switzerland (Poster), 24–26th May.
in synchronization. J. Motor Behav. 29, 35–46. Miyake, I., 1902. Researches on rhythmic action. Stud. Yale
¨
Aschersleben, G., Musseler, J., 1999. Dissociations in the Psychol. Lab. 10, 1–48.
timing of stationary and moving stimuli. J. Exp. Psychol. Radeau, M., 1985. Signal intensity, task context and auditory–
Hum. 25, 1709–1720. visual interaction. Perception 14, 571–577.
Bertelson, P., 1999. Ventriloquism: a case of cross-modal Radeau, M., Bertelson, P., 1987. Auditory–visual interaction
perceptual grouping. In: Aschersleben, G., Bachmann, T., and the timing of inputs: Thomas (1941) revisited. Psychol.
¨
Musseler, J. (Eds.), Cognitive Contributions to the Percep- Res. 49, 17–22.
tion of Spatial and Temporal Events. Elsevier, Amsterdam, Spence, C., Shore, D.I., Klein, R.M., 2001. Multisensory prior
pp. 347–362. entry. J. Exp. Psychol. Gen. 130, 799–832.
Bertelson, P., Aschersleben, G. (in press) Temporal ventrilo- Stevens, L.T., 1886. On the time sense. Mind 11, 393–404.
quism: Crossmodal interaction on the time dimension. 1) Vos, P.G., Mates, J., van Kruysbergen, N.W., 1995. The
Evidence from auditory-visual time order judgments. Int. J. perceptual centre of a stimulus as the cue for synchronization
Psychophysiol. 50, 147–155. to a metronome. Q. J. Exp. Psychol.-A 48, 1024–1040.
Bertelson, P., Radeau, M., 1981. Cross-modal bias and percep- Welch, R.B., DuttonHart, I.D., Warren, D.H., 1986. Contribu-
tual fusion with auditory–visual spatial discordance. Percept. tions of audition and vision to temporal rate perception.
Psychophys. 29, 578–587. Percept. Psychophys. 39, 294–300.You can also read
