|
Ecomedia
How the Natural World is Transforming the Nature of Media.
This weblog is not the usual blog of daily events. It contains a series of notes/thoughts designed to make connections between science and media art.
Sometimes these ideas are tied in with current events, but most of the time this blog is not in any particular order. It serves as a central area for a detailed examination of ideas first published in a 1999 Leonardo Journal article entitled 'Active Vision' that I hope to develop into a book that will discuss some of the current developments in science, ecology, media and society and how they inform and are informed by new technologies. The book will be written for artists working with digital media and anyone who is interested in future directions of the medium.
http://www.andreapolli.com
last modified Sep 7, 2006 at 13:00
During the last solar eclipse, I was teaching a photography class. Of course the class had made sets of cards with pinholes to allow viewing of the eclipse without looking at it directly, and when the time came, we headed ouside to have a look.
We stood on the grounds of the school as the moon started to block the light of the sun and when it was about half way, I noticed something very strange and beautiful. We were standing under a group of trees full of leaves since it was springtime, and what I noticed was a scattering of tiny crescent shapes of light covering the ground, trunks of the trees, and our bodies. The intersections of the leaves in the dense foliage were acting like tiny pinholes and creating virtual images of the eclipse where normally we would see round points of light. As the eclipse progressed, the thousands of tiny images of the eclipse became sharper until they finally disappeared.
From the perpective of Media Ecology, that is: the role that media environments, technology, techniques, and codes play in human lives, data visualization and sonification are changing the landscape. Yesterday I spoke with a retired New York City high school math teacher, and she described how the technology of visualization has created a major shift in the teaching and learning of calculus. High school students in advanced math courses are required to learn using a state of the art graphing calculator, because the calculators are required on a nationwide college advanced placement test. This calculator, unlike previous technology, is able to graph advanced functions in calculus, allowing students to see visualizations never before accessible to them. Because of the calculator, the questions on the national exam have become more difficult to exploit the visualizing power of the calculator. The technology is changing the way students look at (literally) and understand calculus.
The National Science Foundation (NSF) is a large organization in the US that gives grants for scientific research. In recent years, the NSF has required research proposals to include a plan for visualization. The power of scientific visualization has become so important, that new research without a visualization plan is less highly regarded by the scientific community represented by the NSF.
How will the emphasis on the interpretation of scientific information through media affect scientific research? Will scientists and the general public confuse the mediated image with the actual science and therefore understand an only one-dimensional projection of the science? I spoke to two artists working in a nanotechnology lab in Quebec. They were working with a researcher to create visualizations of atoms and molecules, and said that they were surprised to learn that visualizations of objects at that scale cannot be obtained via any mechanical photographic process. The objects are too small for a conventional camera, and even too small for an electron microscope, so the visualizations are created through a labor intensive process of recording and interpreting data. However, the images they create read like an actual photograph.
Although visualization of information can be extremely useful to scientists, it is important to realize that not all information is contained in the visualization. Visualizations of nanotechnology, although they appear photographic, are not the same as actual photographic representations.
Soundscape composer Murray Schafer was the first composer of the modern era to formalize soundscape terminology. He defined background sounds as 'keynotes', foreground sounds as 'sound signals' and called sounds regarded as particularly important to a community 'soundmarks' named from the term landmark.
Schafer believed that the modern era was dominated by 'eye culture' and that listening skills (in particular the ability to identify a location based on sound) had been lost in eye dominated society. although I don't completely agree with the idea that listening skills have been lost in the modern era (recorded music and broadcast sound is unavoidable and contributes to a rich, although perhaps unwanted, soundscape of the modern urban dweller), there is a problem in that soundscapes are evolving and disappearing so quickly that the phenomenon of the 'soundmark' may be unknown to many people. Instead of a sound being associated with a location for many years, there are phases of sounds through communities as illustrated by Andra McCartney's ongoing sonic investigation of the LaChine Canal in Montreal. In her series of soundwalks, listeners can hear an evolution of the canal, hearing seasons of construction, landscaping, and human and automated traffic.
Why is scientific data so often presented as visual information through graphs and visualizations and much less often presented as sound? One reason might have to do with time. As has been discussed earlier, vision is a temporal process. A still visual image can be scanned over time, allowing a viewer to study various aspects of an image. A. L. Yarbus in investigating eye movements determined that viewers looking at a still image will examine different aspects of the image when they have different goals in mind. A soundscape or piece of music, although it is also temporal, cannot be examined in detail without the destructive process of stopping, selecting, and replaying various parts.
Aspects of the visual image are also easily defined by viewers. Specific colors and shapes can be described and understood more often than specific notes or musical phrases. Specific sounds also can have a level of ambiguity. Although some sounds are easily identified (like a barking dog or a cat's meow for example) the source of other sounds are not quite as clear. In my neighborhood, the air is often pierced by various machine sounds, and often it is unclear of what kind of machine or process is creating the sound. If an echo interacts with the sound, it is like wearing glasses that are heavily fogged, making recoognition more difficult.
However, unlike a still visual image, soundscapes and music are inherently narrative. As I listen to footsteps and voices outside my door, I can determine that two people are walking up the stairs of my apartment building. I can determine approximately what floor they are on and even a little about their relationship (are they a couple? a mother and child? have they been arguing or laughing?) In a visual image, a photograph of a family for example, unless the emotional states of the subjects are highly exaggerated, the observer is likely to find a certain amount of ambiguity in determining the relationships between the subjects.
Can an enhanced narrative and emotional content enhance the understanding of scientific data? Some meteorologists call themselves 'storm hunters'. They will travel far and wide at considerable physical risk in order to experience a hurricane or tornado. Does this physical experience that comes with an emotional exhileration enhance the scientist's understanding of the storm? The storms hunters would most certainly say yes. They experience the sound, scale, and physical proerties of the storm as well as its effect on the environment. A storm experienced only through a visualization, whether animated or static, does not convey any of this visceral information. Scientists must use their imagination to create a mental image of the storm's devastation. Does a scientist's visceral sense of a storm benefit the community? Certainly if a scientist can feel if a storm is likely to cause destruction or not, communities can be notified and prepare a response to the storm.
A sonification can create an emotional, visceral experience unlike the experience of a visualization. This experience can alert the scientist to potential dangers or anomalies and promote a more detailed examination of various aspects of any kind of scientific data.
In 1787, the German musician and physicist Ernst Chladni published 'Discovery of the Theory of Pitch.' In it, he describes experiments with sound vibration. He took metal plates covered with a fine grained sand and sent sound vibrations through the plates. Using this technique, he was able to analyze images of the sound of the violin versus the sound of other instruments. Chladni is known as the father of modern acoustics and his work formed the basis for the development of the oscilloscope and computer based sound modeling systems.
In my opinion, a significant aspect of Chladni's work should be re-examined by the digital media community. The oscilloscope and other screen based acoustic analysis systems present the sound waves on a 2-dimensional surface. Practically, this is effective since the standard computer interface uses a 2-dimensional flat screen. However, Chladni's initial experiments were also intriguing because his representations of sound using sand was 3-dimensional. Propogating sound waves surround us. We are literally submerged (or immersed) in these vibrations. Chladni translated these sounds into moving sculptures through his work, showing the 3-dimensional patterns of the sound waves.
In creating translations of weather data into surround sound environments, I have found that it is possible to create the experience of being immersed in the weather system. When listening to the sounds, my collaborator Dr. Glenn Van Knowe, a meteorological scientist at MESO, has found that he can hear something he calls 'gravity waves', a weather phenomenon in which the atmosphere creates a series of waves in the air. Invisible to the eye, these waves are detectable in the weather data gathered and behave like sound waves moving through space.
In fact, waves created in the air by pressure changes do create sound, the whistling of wind harmonics or the low rumble of thunder.
If imagery that surrounds the viewer is essential for the experience of immersion, then the experience of immersion depends somewhat on the human visual system. An understanding of the field of view, perception of motion, stereoscopic vision, and eye movements can improve the design of head mounted virtual reality systems. Objects that appear in the periphery of vision can be rendered at a lower resolution than those in the center of the field of view, but movement in the periphery of vision should not be sacrificed as the human visual system perceives motion with more acuity on the periphery.
Virtual spaces could be used to demonstrate the perceptual systems of other creatures (simulating insect vision, for example) but the experience in a virtual space must always be mediated by the human perceptual system and therefore always will be a mere approximation.
Considering peripheral vision in relation to the center of vision always brings me to a consideration of attention. Research has shown that humans attend to something before their eyes move to focus on it. Right now, as I am sitting at my computer, I can experience this through a little experiment. I move my eyes to focus on a pen sitting on my desk. But, instead of focusing my attention on the pen and without moving my center of focus, I shift my attention to a pile of books directly to my right in my peripheral vision. Then, keeping my eyes fixated on the pen, I shift attention to a printer in the far left of my peripheral vision. I can continue along those lines, never moving my eyes off of the pen, yet attending to every object in the room within my field of view. Through this experiment, I can experience the low resolution of peripheral vision. Although I can attend to a patterned cloth on a table near my writing desk, I cannot examine the pattern on the cloth in detail without shifting it to the center of my vision.
This experience of attention to me implies a disconnect between our visual system and mind. Attention is informed by the visual but not quite directly dictated by the center of focus.
In 'Rethinking VR: Key Concepts and Concerns' by immersive virtual environment artist Char Davies (she prefers the terms 'immersive virtual space' or immersive virtual environment' to 'virtual reality'), she discusses the idea of immersion in relation to her work and other works which envelop the body and serve to enhance the experience of a virtual space. She likens the experience of immersive virtual space to the experience of a scuba diver, immersed in the ocean.
One interesting observation she makes is that she has not found that the sharpness of an image enhances the experience of immersion. In fact, she has found that the use of high resolution graphics actually decreases the experience of immersion for the observer. What, then, increases the effect of immersion? Davies believes that the use of the head mounted display that surrounds the viewer and minimizes distractions is important. She also focuses on the mode of interaction, emphasizing that the use of a full-body interaction enhances the feeling of 'place.' (navigation in her work has been based on the user's breathing for example)
It is essential to her that visitors to her immersive virtual environments experience this environment as an actual place. The interesting part of this argument is that in order for a virtual environment to be experienced as a place, it must be experienced through the 'inside,' that is, through the body of the visitor. Imagery and sound must fully surround and connect seamlessly with the body of the visitor. Like with actual reality, the subjective experience of the viewer is the only possible experience.
How can such spaces be edifying to the visitor? Davies believes that immersive virtual space can be a space where mental models and abstract constructs of the world can be experienced. Concepts of abstract physics, for example, of a very small or very large scale where it is theorized that physical laws are strikingly different than human scale physical experience are being explored through virtual reality systems. Virtual immersive environments can also be used to explore abstract data spaces. Large, multi-dimensional, complex data sets could become familiar locations to researchers and allow for faster identification of patterns and changes in complex systems.
Char Davies ' Rethinking VR: Key Concepts and Concerns' The Proceedings of the Ninth Annual Conference on Virtual Systems and Multimedia 2003
There is a developing movement of artist, technologists, and scientists working with what is currently called 'locative media.' The term refers to media integrated with location information. Most of the work is done using gps data, but projects using mapping processes not automated by the computer are also identified as locative.
Some examples are: http://www.londonbloggers.co.uk, http://murmure.ca/, http://www.blogmapper.com/, and http://www.purselipsquarejaw.org/, http://www.gpster.net/
Mark Tuters, the chair of a panel I just saw as part of the VSMM conference in Montreal http://www.vsmm.org spoke about the natural relationship between gps and cel phone technology and the potential of the cel phone to bridge the digital divide.
The panel, consisting of Anne Galloway, Karlis Kalnins, Jason Harlon and the collaborative artist team that produced the 'murmer' project seemed to me to envision a future in which locative media is ubiquitous. Tuters spoke of new models for social interaction, one he presented was inspired by ant communication. Ants send messages to one another by leaving a scent trail of phermones. Other ants pick up on the trail and can follow it to the location of food, shelter, etc. In the book 'Emergence' by Stephen Johnson, the model of the ant colony is central to his discussion of emergent behavior, and Tuters compared ant gathering behavior to the behavior of the 'flash mobs', the forming of instant gatherings through cel phone text messaging.
I had just finished my own presentation, in which I had talked a bit about the phenomenon of sensitive dependence on initial conditions present in chaotic systems, and I used the classic example of the 'butterfly effect' coined by meterologist Edward Lorenz in the 1960's. I started thinking about these insects, these ants communicating via phermone trails encouraging other ants to gather, and the butterflies, who through flapping their wings can change weather patterns on the other side of the globe, and thisn starting me thinking about the relationship of locative media to the weather.
If people began to carry cel phones equipped with gps on a large scale (and, as Johnson says in Emergence, 'more is different'), and if this gps information was accessible to meteorologists, would this information be able to be used in developing more accurate weather prediction? In a sense, it would be like knowing about the flapping wings of each butterfly, which, if known, could be incorporated into a weather model. Now you may think that it is far fetched to think that cel phones would be that ubiquitous, but if you look at japan, for example, 60% of all people (including babies and older people) carry a cel phone. You may also argue that information like human movement would never be taken into account in meteorological models, but in fact whether a location is urban or rural for example, is currently a part of meteological modeling, a mass of concrete and asphalt affecting the weather differently than an open field. Meteorological models also take into account the chemical makeup of the air, often strongly affected by the presence of industry.
Gisela Leibold was stricken by a stroke that damaged a very specific pathway in her brain. After the stroke, she experienced a strange and rare disorder. She was unable to see motion.
In a crowd of people, she became panicked and disoriented, seeing people disappear and appear in a different location. Riding an escalator or crossing a street was terrifying to her, and pouring a cup of coffee was almost impossible as she would see the stream of coffee entering the cup as a solid, motionless shape.
From Seeing, Hearing, and Smelling the World A Report from the Howard Hughes Medical Institute 'How We See Things That Move: The Strange Symptoms of Blindness to Motion'
While it can be argued that certain fine artists have insisted that the creative process must occur outside of the isolated studio setting in the 'real world,' it is interesting that like experimental scientists in their laboratories, media artists have developed a specialized isolated space for media production. The studio, soundstage, or film set all are are attempts to create a controlled depiction of reality, or more accurately the filmmaker or media artist's imaginative take on the real world. The invention of photography, with all its technical requirements and limitations, brought artists into the studio (and the darkroom) as a place of experimentation under controlled conditions.
As early as Marey, we see controlled, specialized spaces that allow artists to isolate very specific aspects of the visual image, and with the development of moving pictures and recorded sound, these spaces become even more specialized. Although first the design of these soundstage studios were based on the stage of live performance, unlike a stage, media production spaces are only rarely spaces of public interaction.
The invention of the computer and its development as a media tool brought the media artist even further into isolation from the real world. Sets, effects, props, and even characters are created fully in the virtual studio space of the computer, re-creating a version of reality that is simplified for the purposes of representation. Like the laboratory, this controlled space of representation allows media artists complete control of the representation. However, for the representation to be convincing and effective, the real world is often required. For example, in the recent Lord of the Rings trilogy, a primary character is a fully CG character of the Golem. However, the movements of the Golem and his interaction with other characters are controlled by a human performer. The actions of this performer are motion tracked directly into the actions of the CG character. According to director Peter Jackson, the Golem's role in the film series was expanded because of the talent and charisma of the human performer. This 'real world' activity (although isolated on the studio set it could be considered more 'real' than the virtual studio on the computer) had an impact on the vision of the filmmaker.
In the previous post, I mentioned that the experimental sciences began to dominate the scientific method in the 17th century, however, it wasn't until the 19th century that experimental psychology had its formal beginning. Gustav Theodor Fechner was a scientists trained in medicine who through an interest in physics and mathematics became a Professor of Physics at Leipzig University. In 1840, he presented an article on subjective afterimages and that same year suffered temporary blindness from staring at the sun during his research. He left Leipzig for eight years and began a study of metaphysics, returning to the University as a Professor of Philosophy. Between 1851 and 1860 Fechner developed an experimental method for measuring sensation and used this method as a rationale for a philosophical argument for mind and matter unity. Although his philosophical ideas were largely ignored, his experimental process became the basis for experimental psychology. His work was followed by the pioneering work experimental psychology work of Helmholtz and Donders, discussed in a previous post.
Robert H. Wozniak Mind and Body: Rene Descartes to William James. Serendip Online Magazine, Bryn Mawr http://serendip.brynmawr.edu
According to Johannes M. Zanker and Jochen Zeil of the Visual Science Group, Research School of Biological Sciences at Australian National University, several factors have made it possible for scientific researchers to move out of the controlled environment of the laboratory and begin to study perceptual systems in the field, allowing for a look at perception under real-world conditions.
These factors are:
Technological innovation in the area of temporal and spatial high resolution portable recording devices, new theoretical approaches that involve the analysis of complex systems, new knowledge of neurophysiology and the ability to monitor and record nerve cell activity, and, advances in robotic technology that allows the systems to be put to test interacting with the real world.
Zanker and Zeil believe that studying perception in the real world has had and will continue to have a huge impact on scientists' ability to study and understand human motion vision. They see a theortetical understanding of complex systems as essential to the process.
Historically, most of the study of vision and perception has been done through analysis of observation in the real world. However, since the 17th century when experimental sciences began to dominate the scientific method, experimenters from all areas moved into the laboratory. This constrained system allowed for a detailed empirical study of specific process and accelerated the development of science.
What Zanker and Zeil are now arguing, is that the development in technology combined with a theory of complex systems now makes it possible for experimenters to return to the field and combine the best of both worlds: interaction in the real world with the detailed analysis possible in the laboratory.
Scientists moving back into the real world armed with knowledge gained in the laboratory opens the door for more interaction between the sciences and the arts. Although some artists choose to isolate themselves in the studio for specific ends, most artists work and live in the real world. For artists, this interaction is essential to the artistic process, and in fact many contemporary artists have gone so far as to as to refuse to separate art from life (Fluxus, Ukeles and environmental artists, Chris Burden, etc.), making a strong statement about art as a real-world process.
In collaboration, scientists can bring new tools and knowledge of complex systems to the field, while artists can bring a working process of observation and analysis that never abandoned the real world for the laboratory. This is creating individual scientist/artists hybrids similar to historical figures like Goethe and Descartes, but with new experimental and experiential knowledge.
Processing Motion in the Real World by Johannes M. Zanker and Jochen Zeil of the Visual Science Group, Research School of Biological Sciences at Australian National University from Motion Vision - Computational, Neural, and Ecological Constraints. Springer - Verlag, New York 2001
Designers of computer vision often look to biology for models of ways to design vision systems. The developing visual system in human infants is known to progress through several stages. At birth, humans can orient their eyes and head toward a single visual stimulus (usually a human face). By 3 months, most babies have developed the ability to switch visual attention between several stimuli, using a combination of saccadic and smooth pursuit movement. By 5-6 months, the ability to combine reaching a grasping with visual attention is developed, and by approximately 12 months, most bables have developed the ability to use visual stimulus to control locomotion, that is, they walk.
The fully developed human eye has more receptors at the center of the retina, allowing for higher resolution 'in-focus' viewing at the center of attention and lower resolution at the periphery. One unusual finding in human infant development is that newborns to not have this concentration of receptors, but they have been found to bring objects of attention to the adult center of vision. This speaks to me as an argument for the presence of an integrated sensory system in humans, a combination of smell, hearing, touch, and vision all contributing to frontal orientation and attention.
Gaze Control: A Developmental Perspective by Jane Atkinson and Oliver Braddick, Visual Development Unit, University College London from Motion Vision - Computational, Neural, and Ecological Contraints Springer-Verlag New York 2001
As I look out my office window at the Queensboro Bridge, a bridge that I look at and listen to every day, and that I travel over almost every day either by foot, bicycle, or train, I come to realize how the bridge has become a kind of archetype for me.
The Queensboro Bridge, completed in 1908 by the bridge engineer Gustav Lindenthal in collaboration with architects Leffert L. Buck and Henry Hornbostel, is one of the most beautiful cantilever bridges ever built. Although it is made entirely of steel, the bridge appears to be delicate and intricate, like finely stitched lace.
The brige has two large cantilever spans of 1,182 feet and 984 feet connected by a smaller span of 630 feet over Blackwell's Island. The total length of the bridge is 7,449 feet with four 350 foot steel towers constructed on stone piers. The bridge sits 130 feet over the water and contains 50,000 tons of steel.
During the construction of the bridge, a windstorm destroyed a part of the incomplete structure and threatened to halt the project. All told, the bridge cost $20 million and cost 50 lives.
The bridge transformed Queens from a rural community to a borough populated by millions. In its early years, the bridge was seen as a symbol of wealth and elegance, it figures prominently in F. Scott Fitzgerald's The Great Gatsby as the Manhattan connection for wealthy Long Islanders.
In the 1950's the bridge started to show signs of decay from years of neglect. The tall spires on the top of the towers were so corroded they had to be removed. In 1974, the bridge was designated as a New york City Landmark, and recently, in the 1990's the New York Department of Transportation invested almost $300 million in the reconstruction of aspects of the bridge.
Since 1982, only two people have died from jumping or falling off the bridge, out of 8 documented cases of jumping.
The bridge has become a big part of my life. It is my lifeline and I depend on it to get to Manhattan and back home again. It both represents and is a connection. It gives me a bird's eye view of my environs, an experience I find both exhilerating and relaxing. It is both a destination and a way to reach a destination.
One of the more dramatic yet unexpected experiences of the bridge I have had is the experience of the enormous variety of sounds that are generated by the traffic on the bridge. Two train lines cross and turn on the Queens side of the bridge, and the combination of these high pitched squeals and rumbles create a tremendous cacophony. The contrant stream of cars on the bridge, never stopping either day or night, creates a constant low rumble.
|