(6-1) Fusion of sight and
Dodd and Y. Sakurai: "Localization of sound\A New Zealand Revelationh, NZ
Acoustics, vol16,no4, p28|33
Contradiction on the law of the first wave front
When a movie was projected on a large
screen in front, and stereophonic reproduction was given from two loudspeakers
in behind, the localization of the sound was happened at the screen. All of 7
or 8 people who remained after the movie gathering answered that they heard the
sound from the screen as well. It contradicts the first wave front law (the Haas
This phenomenon did not happen, when slides
were projected on the screen and sound was given from the same place as the
movie was played. The sound was heard from the loudspeakers behind.
On a different day, a movie was played on a
smaller screen of the half size. It was recognized that sound was heard from
the rear speakers at first. As time passed, though, the sound impression
gradually moved to forward. However, it never came from the screen to the last
Interesting enough is that even after the
screen was turned off, the sound was heard from forward. It is thought that it
was caused by learning or plasticity.
In order to explain this phenomenon more,
the introduction of the mutual reaction of the sight and the audition must be needed.
Generally, could we tend to be attracted to the sight more strongly? Can we say
that the sight is predominant? How each other is different at information
We will have to consider at the sound
evaluation of a concert hall if the sight influences the localization of sound
in this manner. That is, we can not neglect the sight design on the stage
related to the acoustic design.
In addition, it will be involved when the
reflection in a concert hall is evaluated. When we discussed on the acoustical
renovation at Tangle Wood Music Shed, Harold Marshall said that spaciousness
can be obtained even if lateral reflection is given from the rear sides, though
the reason missed being heard.
2) Time and
It is necessary to give a window to take
out necessary information of the object on the sight or the audition which
expands or continues. One is a time window to obtain the information of a certain
time interval and another is a spatial window where a certain necessary range
is chosen in the area. It can be said that the weight of the audition for a spatial
window is given in front being related to the head related transfer function1).
However, it is necessary to research in details of the discussion.
There are a time window and a spatial
window in either system as a strong will to get information in this manner.
3) Fusion of
sight and audition
The time window may be short for a still
picture because information can be confirmed soon. There information magnitude
or intensity is small. The time window is long, and the magnitude of
information is large for a moving image.
This appearance changes depending on the
dimension of an object, the speed of change, the kind of information and etc. It
is necessary to discuss being related to eye-movement.
When a screen is large, eye-movement
becomes great and rapid, and a large effort is necessary for the information
gathering of the sight. Then, the effort is concentrated only on the sight and
the audition processing is forgotten.
Is it possible to have the following
We have the attitude to combine the two
different sensory inputs into one information, if they are of the same source.
These two systems are not good at simultaneous perception for strong
stimulation above a certain level. And, it follows to the sense with the
However, intensity or magnitude on each
system is within certain strength, each response becomes independent.
Examples: It was the case when sight
stimulation was stronger at the movie. The specification of a view is difficult
when a lot of machines produce big noises at the same time. The sight of performers
is not important when one is deep in listening to a wonderful music. When we
hear noise of certain level, we can not concentrate our mind. When we work with
music we lose efficiency. They are cases when audition is occupied stronger
When a movie screen is small, the
eye-movement is small, and each can be separated. However, though the sound is
recognized coming behind at first, the sound image moved forward as time passed.
And it was never heard from the screen. It is said that the sound image existed
in front even right after the screen disappeared. This experience is different
from at the still picture. It could be explained as learning or prasticity.
Now, we have to discuss on the relationship of the stimulation intensity of
both sensation systems. The intensity of each sensation would be multivariable;
change in time, range of the area with information, easiness of information
acquisition, brightness of a screen, and receiving sound level, etc.
When a still screen cuts to appear
suddenly on a large movie screen, do we hear the sound from the rear? It seems
not to be so. As even at the half screen the sound remained in front after the
projection disappeared, we would keep hearing from the screen. This might be a running
or shifting learning effect for the sight as mentioned previously.
Then, is this learnt attitude possible to be
shaken off in consciousness, and to hear the sound to be recognized coming from
the rear? Why it cannot be done? Then, wrong recognition might be happened at
the place where a sight coexisted with audition in this way.
Though the received level to each ear on
the first wave front is slightly different, stereo effect occurs with their
same levels, and the sight induces it to the middle might be a possible
explanation. However, it is not enough to explain the difference between a
still and a moving screen. There the strength of information should be
When a singer has a wireless microphone and the sound is generated
from the loudspeakers on both sides, does the sound and image attach to the singer's
movement? How about a movie is on the screen? It seems that the stereo effect
is caused at this time, and the sound and image follows easily. Though, the discussion
here is the case when the loudspeakers are placed in the rear.
The diffraction wave chiefly brings low
frequency range sound, and reflection wave composes high frequency range sound
which gives the directivity from the front. This is only an explanation from
the acoustical side and does not explain the different impression of a moving
and still screen.
5) Experiments in
Two dummy loud-speakers are displayed at
each side at the screen. Stereo sets of loud-speakers are distributed along the
bottom corners of each side wall, and ask them where sound is coming from
showing a movie. Eye-movement measurement should be done together.
Sight; still or moving, large and small,
change of speed
Sound source; stereo or monaural,
Optics; brightness of a screen
It is necessary to have multivariable
analysis with above factors at the preparation. The introduction of the correlation
function between two sensory systems would arise.
In order to concentrate the discussion, a movie of only one language
should be projected.
Do the early reflection and or the sound diffusion influence? If so,
spaciousness should be discussed on the subject.
When we close our eyes at the movie where does the sound come from?
How will it be different when sound is for
the screen and when it is not related to the screen. How about when the sound
is music? Does our mind move to the screen especially even when the music is
not related to it?
When we discuss on learning effect, we
have to talk about memory. However, the phenomena do not look to happen at the
central but at peripheral. Namely it is the process before the cerebrum to
understand. It happened as well when the subtitle was too quick and hard for me
1) My homepage at Chapter 6 (3-1)
Acoustical evaluation of an auditorium
Peep into visual sensory system
Its nature and behavior
When a movie stops and we see a stopped
frame, it looses clear resolution. Such discrete frames are connected with our
spectrum but actually what is happening in our visual system. There they must
have character to connect them up. The visual system also takes
auto-correlation of a signal and leaves the remaining effect that was measured
at (5-1) of this section.
It also estimates the next from a
moving object. It has an attitude to connect in time.
When we see an object with naked eyes it
looks natural. But if we see a pair of parallax pictures through a stereoscopic
viewer, the object looks like painting on a sheet of paper, no depth. Namely,
the pair is fixed and no information exchange in-between occurs. The former
gets natural 3D with slight eye movements.
The system does not want to have
discontinuity in space or is impossible to do so. Is it imagined with the cross
correlation between both eyes? It looses some information because of the
process and needs to supply it with learning and experience.
It has an attitude to connect spatial
discontinuity with the mutual talk of both eyes. Visual system looks for
correlations spatially and temporally. There are no information split into
pieces and it tries to connect smoothly.
For an example, at the border
between black and white, each side luminance is not discontinuous but smoothly
connected referring to the effective luminance theory. There exists learning in
The speed of input visual
information is so fast and it has to be judged clearly with short time, the
concentration to processing it needs a lot of work and efforts. Accordingly,
the visual processing precedes the audition system. Toward visual information,
we have basically an attitude on a visual object to fix it and search it.
Formation of cells for particular purpose and
When a baby is born, he
can not recognize anything. Licking his fingers he gets a visual cell to
recognize his hand. He does not count the number of fingers to recognize it. He
sees his motherfs face always in front of him and gets a visual cell to
recognize a face.
During this period, he grows a visual cell with which he can
distinguish front or behind. Step by step he always learns to establish his
We can not work by hand without knowing a boundary and depth in a
moment. We have to judge front or behind. We establish the quick recognition
system. There has to exist particular cells to do immediately.
The system reacts so fast and needs a switching mechanism in addition.
It has to react to the light and its transient response must be so fast. It is
never the same of spectrum that lasts 400ms. See the section (5-1) of this
When we see characters on a sheet of paper or scenery, the location of
each cell is different. And it needs switching. Our hearing system has one too,
e.g. party effect. An established cell to see depth exists and obtained
information is exchanged with another eye. Such exchanging or mutual talking
must be established too.
As we have the regulation to luminosity,
we regulate our visual system to read characters on a sheet of paper, to see
near, middle or far distances. We move attention to each area consciously. In
addition, we have a switching to see vaguely and widely
The switching at the regulation to
luminosity is not done by our mind. The mechanism is imprinted. Other switching must be done the same. Its
transform is on the fuzzy system. The concept of effective luminance is a good
down relaxed and enjoying surrounding scenery might be the check up of the
switching system and normalization of the deformed part of the system. To have
stress free must mean this process. When we think about our cells get fast metabolism, it must be related
to this check up not only learning.
Looking up the blue sky, looking over the
wide sea, looking down a wide spreading forest etc all is needed and important.
Other wise we can not have a broad view and mind.
of crystalline lens
The visual system has only eyes to get visual information
and focus to an object. The diffracted light in the crystalline lens widens the
view angle and gives three dimensional data as well. It might start to have the
mutual talk with the other side.
The retina is a plane screen and distributes inside the crystalline
lens. It collects three dimensional data and maps them on the retina. The
retina starts to have the mutual talk with the other side.
There exists a function to process the perspective. Namely, it sends
three dimensional data to the retina and process there to give it higher level.
talk, cross correlation and learning
If we see an object by a single eye, it
looks as if painted on a plate. The distance is felt by two eyes. It is found
out by both eyes. It is not done only by the parallax. They look to have cross
correlation in between and add learning. Although they get a distance with the
tinny difference parallax, it is not enough and they need the mutual talk. This
is the same for hearing system.
They take cross correlation but
get information loss by that. Learning and experience are needed to supplement.
If this process
can not be done fast, it is impossible to grab a thing or carry it to one place
to another on a daily life.
The mutual talk between two ears is not only done by them but also
through bone conduction and skin. On the other hand, the visual system does it
only by the sight and cleverly maps it.
After all, an object is sensed
front or behind by corresponding cells through an eye. Both eyes get the mutual
information exchange to have distance, however, it is not sure yet and get
quick eye movements to make it sure.
The visual and hearing systems
have commonly these movements to make it sure. A 3D screen is helped by a moving object
where the visual system works to understand. It is not real simulation.
of sight and audition and their comparison
Why a stereo system does create a sound
image in front? The reason could be that there are visual things in front. Even
if lateral reflections come from
behind, they give still spaciousness. Visual objects are on the
stage and the same fusion of two systems happens there.
visual system, a moving object gives a lot of information to fix it. It looks
to be easier to do so.
It must be
quick to have the direct connection to judge if it is dangerous or not. It is
essential to have the function for immediate decision.
How about the
mutual talk between two ears? We do not fix our head at a certain point but we
move it always slightly. It looks to have this slight movement on both visual
and hearing systems.
In the mutual
talk level, the visual system responds to the light speed and the hearing
system to the sound speed. Visual processing is done much faster than hearing
There was a
person at the movie gathering who felt the sound was from the rear when he
moved to front close to the screen. There must be a few conditions to have the
fusion. In his case, the sound level from the rear loudspeakers were directly
reached him and loud.
localize a sound source which is out of the focal plane, especially behind, we
hear it through ears and with bone conduction but through skin as well.
response of hearing system, temperature sensation and visual system lasts in
5ms, 150min. 400ms respectively. They are given at (5-1) of this chapter. 400ms
is for the visual system to give spectrum.
A little additional comment
If the imprinting
process of learning and experience is given in a concrete box, it would be
unnatural and does not adapt to nature well.
A rock singer
was playing with an electrical guitar on the stage in front of two
loudspeakers. Where is the sound image created, in the middle of the
loudspeakers or does it move with him? It is interesting to see with
It is very difficult to set a
microphone at a given spot in anechoic chamber where there is no reference near
by. In such a case, visual intuition helps a lot.
A few interesting
fusions of sensors: the sense of touch and visual system. Smell, look and