|
Ecomedia
How the Natural World is Transforming the Nature of Media.
This weblog is not the usual blog of daily events. It contains a series of notes/thoughts designed to make connections between science and media art.
Sometimes these ideas are tied in with current events, but most of the time this blog is not in any particular order. It serves as a central area for a detailed examination of ideas first published in a 1999 Leonardo Journal article entitled 'Active Vision' that I hope to develop into a book that will discuss some of the current developments in science, ecology, media and society and how they inform and are informed by new technologies. The book will be written for artists working with digital media and anyone who is interested in future directions of the medium.
http://www.andreapolli.com
last modified Sep 7, 2006 at 13:00
In the early 1880s Etienne-Jules Marey conducted research in the portrayal of motion using photography. He worked in a laboratory he constructed ouside Paris called the Station Physiologique. His method of moving unexposed film, "chronophotography", allowed him to study the mechanics of motion, and his images attempted to isolate the 'purity' of motion by dressing models in all black with white stripes, dots, and electric lights placed lengthwise along the limbs and at axis points. Marey's costumed models looked very much like contemprary models equipped with motion tracking sensors used for animation and game design.
In an extensive research project studying efficency in the industrial age, Frank Gilbreth and his wife Lillian produced 2,250 glass plate photographic images between 1910 and 1924. These works included various long exposures of workers on assembly lines, operating typewriters, laying brick, involved in basically any working activity of the time. Besides being a great document of working life during the early 20th century, these photographs helped to define photography's role as a scientific research tool.
The technique they created using timed exposure of workers with lights attached to their hands and feet serve to distill motion into a simplified form more easily evaluated for efficiency. This essence of motion, that the Gilbreths broke down into a combination of basic building-block movements, looks very much like motion paths described by contemporary computer animators.
Catching a Glimpse of America's Industrial Past. Michael Kernan Smithsonian Magazine May 1998
Thomas, Ann ed. Beauty of Another Order: Photography in Science Yale University Press: New Haven 1998
Impressionist painters portrayed the movement of light, cubist painters portrayed movement of figures and objects in space, and the furturists attempted to portray the movement of machines. Through the medium of photography, Marey, Muybridge, and others experimented with the portayal of the body in motion and helped to bring about moving picture technology. Animation, film, and video all excel at the flat screen portrayal of subject motion. Observer motion is also modeled in film through precise camera choreography.
Interactive moving image technology presents a unique opportunity to not only portray objects and subjects in motion, but to portray the experience of the observer in motion. In an attempt to create realistic computer gaming experiences and intelligent robot movement, researchers have begun to study how biological creatures are able to perceive and resolve the motion of their bodies.
There is a great variety of living eyes on earth, ranging from the very simple to the extremely complex. What are the simplest living visual systems and can the design of these systems give us a new way of considering our own visual process?
Some simple multi-cellular creatures have certain cells that are more light senstive than others that transmit information to other cells. These cells, concentrated in an area called the 'eye spot' are either flat on the surface of the body, or set in an indented dimple. The curvature of the indented eye, called a 'cup eye' allows the living creature to determine the directionality of the light stimulus. The flatworm has a light sensitive cup eye with cells similar to the human retina.
In other more evolved visual systems, the indentation closes at the edges, creating something like the pinhole of a pinhole camera. The inside of this biological pinhole camera is filled with a protective clear gel-like substace. Like a pinhole camera, there is a balance between the perceived image and the size of the pinhole. The smaller the pinhole, the more detailed the perceived image, however, if the pinhole is too small, not enough light can enter to register an image. Creatures with pinhole eyes include the Nautilus and the marine snail.
Eyes with lenses allow the entry of more light while still focusing the image. Some simple eye lenses are forms of a clear gel substance, others from a mass or transparent cells.
For creatures living in environments that have various light intensities, a pupil regulating the light input is a great benefit to vision. The pupil of an eye doesn't have to be circular like the human eye. Look closely at the eye of a snake or even your cat and you will see pupils that look more like vertical slits than circles. Then look closely at the aperture of your camera and you will see an imperfect circle, a polygon made of many slats.
In The Origin of Species, Charles Darwin discusses the lack of vision in some mammals:
"In South America, a burrowing rodent, the tuco-tuco, or Ctenomys, is even more subterranean in its habits than the mole; and I was assured by a Spaniard, who had often caught them, that they were frequently blind; one which I kept alive was certainly in this condition, the cause, as appeared on dissection, having been inflammation of the nictitating membrane. As frequent inflammation of the eyes must be injurious to any animal, and as eyes are certainly not indispensable to animals with subterranean habits, a reduction in their size with the adhesion of the eyelids and growth of fur over them, might in such case be an advantage; and if so, natural selection would constantly aid the effects of disuse. "
The compound eye of the insect, a kind of eye found in fossilized remains from over 400 million years ago, is very different than the eyes previously discussed. In the compound eye, the eye spot area curved outward rather than inward. The insect eye detects motion more effectively than it can form an image with its hundreds of miniature eye like structures callled ommatidia.
Curiously, Walter J. Gehring of the University of Basel in Switzerland in 1993 and various other researchers since have discovered an insect gene called the 'eyeless' gene. Abnormal activity of this gene can cause a fly to sprout eyes on its wings or legs, creating grotesque creatures with as many as fouteen eyes placed all over their bodies. The eyeless gene is similar to a gene identified in humans and other mammals, and the eyeless gene from a mouse was transplanted in a fly and caused the fly to grow additional fly eyes.
"Darwin will rest easier thanks to flies with eyes on their wings" Elizabeth Finkel as broadcast on ABC's Ockham's Razor on the 12/11/95 http://www.wehi.edu.au/resources/vce_biol_science/articles/finkel3.html
Isabella Chow "The Fly" The Moment: Columbia's Science and Engineering Newspaper
http://www.columbia.edu/cu/moment/v0/040595/fly.html
Dawkins, R. 1994. The eye in a twinkling. Nature 368(April 21):690.
Goldsmith, T. 1990. Optimization, constraint, and history in the evolution of eyes. Quarterly Review of Biology 65(September):281.
Gould, S. 1994. Common pathways of illumination. Natural History 103(December):10.
Harris, W. 1997. Pax-6: Where to be conserved is not conservative. Proceedings of the National Academy of Sciences 94(March 18):2098.
Mestel, R. 1996. Secrets in fly's eye. Discover (July):106.
Nilsson, D. 1989. Vision optics and evolution. Bioscience 39(May):298.
Zuker, C. 1994. On the evolution of eyes: Would you like it simple or compound. Science 265(Aug. 5):742.
On the evolution of photoreceptors and eyes.
Evolutionary Biology 10 (1997), 207-263.
Don Lindsay http://www.cs.colorado.edu/~lindsay/creation/eye_stages.html
Dr. John Maiello Department of Biology Rutgers, The State University of New Jersey Campus at Newark course outline Fall 2003 http://tecn.rutgers.edu/genbio102/genbio102/4EvoEyes.html
Charles Darwin, The Origin of Species. Chapter 5 - Laws of Variation
Science News Online "Eye-opening Gene" by John Travis http://www.sciencenews.org/sn_arc97/5_10_97/bob1.htm
Climbing Mount Improbable, Richard Dawkins, W. H. Norton & Company, Inc.
The human retina is a thin sheet of neural tissue that is like a thin sheet of transparent glass decorated with patterns of blood vessels that are more unique to a specific individual than a fingerprint. The circuits of interconnected neurons in the retina perform a very basic image filtering process or compression, accentuating some aspects of visual input and playing down others resulting in an extraction of the essential features of a visual scene. The neurons in the retina are of various types, some responding to directional motion, others to staionary stimulus or color. Successful artificial retinas mimic the retina through a surface of circuits designed to respond to light levels. Imagine a honecomb-like array of silicon circuts creating a low resolution image made of hexagonal shapes instead of square pixels.
Remarkably, users of these artifical retinas experience many of the same optical illusions experienced by an unaided eye. For example: when looking away after viewing an image for a long time, a user of an artificial retina experiences a negative afterimage and the brightness of an object is perceived by the artificial retina relative to its background color
"The Functional Architecture of the Retina" Scientific American 1998 The Mechanics of Sight
"The Silicon Retina" Scientific American 1998 The Mechanics of Sight
This morning I watched a dragon fly do a delicate dance over a pool of water that had formed on our patio after the last night's rain. Its precise movements betrayed an effective visual system, but how do the visual systems of insects differ from those of humans? There has been an increasing interest in the visual systems of insects by researchers designing computer vision for use in robot navigation. Researchers at The Center for Visual Science at Australian National University in Canberra and the Department of Computer Science at Curtin University in Perth collaborated on a project exploring robot navigation inspired by principles of insect vision.
Unlike our eyes, insect eyes are immobile, making depth perception much more difficult. Insect eyes are also much closer together than human eyes, also impeding depth perception. How, then, was a Dragonfly able to skim the surface of a reflective pool of water without crashing into it? Experiments with locusts by G.K. Wallace in the late 1950's researchers determined that the fast, seemingly chaotic motion exhibited by insects and other invertebrates (skittering crabs for instance) is part of the insect's depth perception. Wallace called a series of head movements performed before movement toward an object, the 'peering' action of the locust used to determine the distance of the object.
References:
Srinivasan, M. V. et. al, "Robot Navigation Inspired by Principles of Insect Vision." Robotics and Autonomous Systems 26 (1999) 203-216
Wallace, G. K. "Visual Scanning in the Desert Locust Schistocerca Gregaria" The Journal of Experimental Biology 36 (1959) 512-515
Dr Vincent Chow and Dr Alan Chow began of Optobionics began clinical trials of the Artificial Silicon Retina (tm) in 2000. The retina is a silicon microchip 2mm in diameter and 25 microns thick (less than the thickness of a human hair) that is impanted under the retina in the eye. The chip contains approximately 5000 microscopic solar cells called "microphotodiodes" that convert light energy into electro-chemical impulses that stimulate existing retinal cells.
The Doheny Eye Institute at the University of Southern California has also developed an artificial retina. Similar to Optobionic's retina, their retina is a small disc implated in the eye that captures visual signals and sends them to the brain as electrical impulses. In this case, however, the visual signals are captured by a small video camera in the eyeglasses of the patient and processed through a microcomputer worn on a belt. The signals are transmitted to the electrode array in the eye, that then stimulates the optic nerve.
The term 'artificial retina' is also used for computer vision systems outside the body, Mitsubishi has has had such an 'intelligent system' for detecting the movement of a game player in development since 1990.
References:
Seeing Is Believing: Hope for the Blind — Newsweek
May 19, 2003
63–64
Help in Sight: Alan and Vincent Chow’s experimental silicon eye chip may help correct a common type of blindness — People Magazine
July 15, 2002
73–74
Vision Quest — Dateline NBC
June 4, 2002
With bionic eye, subjects see light for first time in years — USA Today
May 8, 2002
Front page
Company Sees Success and Funding in Helping to Give Sight to the Blind — Small Times
April 11, 2002
A Chip That Mimics a Retina but Strains for Light — New York Times
August 9, 2001
Circuits section
Chow AY, Pardue MT, Chow VY, Peyman GA, Liang C, Perlman JI, Peachey NS. Implantation of silicon chip microphotodiode arrays into the cat subretinal space. IEEE Transactions on Rehabilitation Engineering, Vol. 9 No 1, March 2001, page 86ff. (Link for IEEE members only)
Peachey NS, Chow AY. Subretinal implantation of semiconductor-based photodiodes: Progress and challenges Journal of Rehabilitation Research and Development Vol. 36 No. 4, October 1999
Peyman GA, Chow AY, Liang C, Chow VY, Perlman JI, Peachey NS. Subretinal semiconductor microphotodiode array. Ophthalmic Surgery and Lasers 1998; 29:234–41
Chow AY, Chow VY. Subretinal electrical stimulation of the rabbit retina. Neuroscience Letters 1997; 225:13–6
The US Department of Energy DOE Science News november 25, 2002 http://www.er.doe.gov/Science_News/feature_articles_2002/November/Artificial_Retina/Artificial-Retina.htm
Humayun MS, de Juan E Jr., Dagnelie G, et al. Visual perception elicited by electrical stimulation of retina in blind humans. Archives of OPhthalmology; vol 114; pages 40-46, 1996
Humayun MS, Propst RH, de Juan E Jr., et al. Bipolar surface electrical stimulation of the vertebrate retina. Archives of Ophthalmology; vol 112; pages 110-116, 1994.
Tokyo, November 18, 1996 -- Mitsubishi Electric Corporation
http://global.mitsubishielectric.com/news/1996/mel0434_b.html
In the New York Times yesterday, there was a tiny article that offered advice from a group of Buddhist Monks to people who suffer from uncontrollable cravings for food. They suggested that a person experiencing a craving try to stop, take a deep breath, and feel the experience of the craving. This would make the person aware that feelings are temporary and allow them to separate from the craving. The advice also stated that it was important to not identify a feeling as 'mine' but instead experience it the same way one might experience external stimulus.
Today walking home over the Queensborough bridge separating Manhattan from Queens, I decided to give the advice a try. I wasn't experiencing any particular craving, but I did want to try to merge external and internal stimuli. The bridge is a good place to try it because the view is stunning and continually changing, and the vehicles on the bridge create a loud rhythmic pattern of whistles, clangs, and squeaks that increase in pitch as you near the top of the bridge and descend in pitch as you descend.
The focus of my experience was really to try to expand sensory perception by considering my body as a part of the environment, and it became difficult at one point as two subway trains crossed over my head screeching loudly, I felt as if I was being engulfed by the steel and motors, but rather than internalizing the feeling, I tried to experience it as part of the present.
Although I'm sure I didn't come close to understanding what the Buddhist Monks were describing, it did provide me with some insight for this blog.
The public nature of this blog (although I have only been writing it for two days now) has made me feel less isolated as I work on this project. Without expecting that anyone is actually reading it besides myself, the fact that it is accessible far beyond my hard drive provides a sense of the blurring of the internal and external. There is also a temporal nature to the project, seeing previous writings as part of one growing work brings the past and the present closer.
Over this summer I visited a colleague at Concordia University, Andra McCartney. She and a group of her students have spent the past three years doing soundwalks around the LaChine Canal in Montreal. Soundwalks were done first formally by sound artist Hildegard Westerkamp. The idea is to walk around an environment (often a lake or a park) with recording equipment, and really listen to the sounds of the environment, and record what you hear, including sounds created by you. Andra McCartney's work has been to focus on the 18 kilometers of the canal in all seasons during a period of the canal's history that is changing drastically. Listening to her recordings gives one a strange feeling of diembodiment, a feeling of both being transported to the time and place of the soundwalk, but also a feeling of a continuity between the present time and the recorded past. Like the canal, which is constantly moving yet always present, the sounds around the canal are unique to a specific moment yet iconic in nature.
Written August 2000 while flying over the archipelagos of Iceland at sunrise. Staying at the Red Lion, an Inn behind a pub with rooms that are named after brands of beer.
During the flight, I experience a desire to release from the boundaries of the shell of the body and become part of the surrounding environment. This desire is unrelated to the type of environment (natural, man made), but instead is related to my state of mind. Is this desire related to the desire that brings one to write, to create art, to study the very large and the very small? (i.e. looking through a microscope as a transportation of the body similar to the experience of looking at a religious icon or a mandala).
My words replay texts read and heard in the past, a memory that has been constructed throughout history. Is accessing this history a way to transcend the limits of the body? A way to become a part of the thoughts and consciousness of the past? Are these ideas always with us, as it is said 'part of the aether' that surrounds us, or the waves of energy that encompass our bodies and need to be decoded like receivers decode radio waves. The sensory experience of the splendor of nature cannot be expressed through the word, although historically there are many examples of attempts. In this case, however, do the words enhance the experience like words might (arguably) enhance the experience of a great work of art? I do not think this is the case, even for the most innocent of minds, natural beauty needs no explanation.
I feel at times when in the throes of an idea that a transformation is possible. A feeling that is at once exhilirating and frightening. To be on the threshold of the limits of the body--dead yet alive. Is it that part of the collective memory is the memory of death? And that this memory bubbles to the surface of the conscious mind in the ecstatic moment? Death itself is the only infinite experience of the body.
How does proprio-ception function here? Bodily knowledge informs emotion (the emotion of a lover's touch for example) but how does it come into play in the out of body fantasy? Is transcending the feeling of the body a way to transcend emotion? A way to experience death? Or is loss of proprioception an unwanted but necessary by-product of transcendental experience?
On what level are these sensory or extra-sensory experiences real and in what sense are they delusional? Isn't any model or belief 'real' insomuch as it allows one to function in the society at large? Radical transformation is not possible on a large scale. (what about the monks in India that control their bodily functions through meditation?) The last frontier after the body is the spiritual--but these two are not separate entities, rather they are inextricably linked--in fact function as one--like the earth's core and the surface of the earth.
How has the mass migration of ideas sped up this process? As humans are moving more (more % human movement on the earth has occurred in the last ten years than in the entire history of humankind) Bombarded by images, we experience an infinity of lifetimes through the image. 'Visual search has no memory' but the visual form becomes locked in memory. How does this relate to eye movements--if at all?
Alchemical processes of understanding the body's relationship with the earth and the environment: the earth itself acts metaphorically as a body. The earth's ridges are 'the wound that never heals.' It is significant that images of the spiritual and the metaphysical are often made of the earth and of the moment. Geologically, the langorous pace of the earth is one which is dificult for the human body to understand. In experience, however, the human experence of the earth is one of constant change. Storms suddenly create drastic fluctuation in the visual, aural, and tactile experience of the earth. There is a constance to these fluctuations, however, as humans can experience the cycle of change, a constant re-assurance of the stability of the earth. As you look at long spans of time--again the experence is like looking a large distances--one feels that it is possible to transcend the limitations of a human life span.
One of the main motivations behind the development of the Intuitive Ocusonics system was the desire to create a more direct link between thought and action. In other words, to find out if playing a musical instrument with eye movements would feel like creating music through thought alone. There is a history of research of this kind. In 1990, David Rosenboom published a series of articles about musical instruments and performances made by severeal composers using signals from the human nervous system (see Rosenboom reference).
In the six years since the creation of the Intuitive Ocusonics instument, and in several dozen live performances, I have to confess that only once did I have a feeling of a direc t thought/action connection. I was rehearsing for an improvisational performance with bassist George Cremaschi at CESTA in Tabor, the Czech Republic. At one point during our rehearsal, I actually thought that I wanted to change the sound I was creating and was suprised to hear the sound change to exactly what I wanted without me making a conscious movement. Unfortunately, my suprise immediately shook me out of the moment and I finished the rehearsal making only conscious decisions to change sounds.
For a little while, I thought that this was a significant experience, perhaps an indication that the Intuitive Ocusonics system was indeed experientially closer to thought, but then I realized that any musician deeply familiar with his or her instrument, be it the voice, or guitar, or invented instrument, will have experiences exactly like mine, where the physical requirements of playing the instrument become unconscious and the music is created by what feels like pure intuition.
While speaking to Glenn on the phone yesterday (Glenn Van Knowe, my collaborator in the Atmospherics project, lives in upstate New York. We have regular phone conversations about our project, and many of these will be referenced in this blog), he mentioned a new development in meteorological modeling in which an expert system is used to 'learn' common weather patterns. The goal is to develop more accurate weather prediction, but the current state of the system often ends up with more wrong predictions than the existing physics-based models. Glenn thinks that as the expert system is given more and more weather data to interpret, the system will get better at prediction.
Is this the beginning of a kind of computer intuition? As human meteorologists gain experience in making predictions (or doctors gain experience in diagnosing illnesses), the predictions get better. Often without being able to explain how one knows, an experienced person can make accurate predictions purely on 'intuition' also defined as a 'feeling.'
I brought two friends with me to the microcontroller workshop, and we are going to be getting together to try to build some of the examples and come up with new projects and challenges. Working alongside two friends who experienced the same workshop, but based on past experiences were able to get different information from the workshop, will make the work move faster and increase our ability to create. We jokingly called it our 'brain trust'. Are those who build expert systems attempting to create a 'brain trust' for themselves to use as 'an instrument to achieve meaning that cannot be produced by the body's natural means' ? (Ponty)
Merleau Ponty's words: "Sometimes, finally, the meaning aimed at cannot be achieved by the body’s natural means; it must then build itself an instrument, and it projects thereby around itself a cultural world."
make me think about yesterday's workshop and other experiences I have had in the programming world. During the workshop, one of the participants asked if it would be possible to build a microcontroller that could program itself. Certainly the self-modifying code would be possible, but very impractical for the functionality of the chip. However, the question and the interest in the question that followed indicated to me a mirroring process between humans and our computers. Since many people now spend a majority of waking hours somehow interacting with computers, and for many of us, a lot of that time is spent programming the computer, therefore a natural desire (if one sees the computer itself as a representation or mirror of the self) is to build a computer that can program itself. Related to this are research initiatives to build computers that can create visual design, design electrical circuits, and design and build robots.
Last week an exhibition called 'Artbots' opened at Eyebeam in New York. "ArtBots: The Robot Talent Show is an annual international art exhibition for robotic art and art-making robots." The 2003 show was curated and produced by Douglas Repetto, Philip Galanter, and Jenny Lee. Specific to this discussion is the idea of the "art-making robot" The idea of a machine that can create art can be traced back to Jean Tinguely's Dadaist art making machines. In a very clear act of self-mirroring, Harold Cohen (currently professor of art at UCSD) built Aaron in the early 1980's, an artificial intelligence system designed to draw. Aaron has been growing and developing ever since, and has learned to draw plants, figure, and cityscapes and has also learned to add color to the drawings.
Like Aaron, some contemporary projects in robotics involve the creation of robots that learn over time. Of particular interest are computer vision systems that need to 'see' many images over time to learn to identify forms and objects.
In The Phenomenology of Perception, Merleau Ponty states:
"In so far as I have hands, feet, a body, I sustain around me intentions which are not dependent upon my decisions and which affect my surroundings in a way which I do not choose. These intentions are general ... they originate from other than myself, and I am not surprised to find them in all psycho-physical subjects organized as I am."
"The body is our general medium for having a world. Sometimes it is restricted to the actions necessary for the conservation of life, and accordingly it posits around us a biological world; at other times, elaborating upon these primary actions and moving from their literal to a figurative meaning, it manifests through them a core of new significance: this is true of motor habits [sic] such as dancing. Sometimes, finally, the meaning aimed at cannot be achieved by the body’s natural means; it must then build itself an instrument, and it projects thereby around itself a cultural world."
"The analysis of motor habit as an extension of existence leads ... to an analysis of perceptual habit as the coming into possession of a world. Conversely, every perceptual habit is still a motor habit and here equally the process of grasping a meaning is performed by the body."
From Luciano Da Fontoura Costa's Minding Vision, Oct 1996 about the relationship of vision to consciousness:
"Furthermore, the peculiarities of such a flow of thoughts bear some remarkable analogies with visual awareness. After pursuing some ideas for some time, our thoughts are directed somewhere else, into a new subject that is nonetheless associated in some way to what we were thinking before. If you close your eyes now, it is very likely that you will continue, for some time, thinking about the ideas presented in the last sentences. Then, you may start wondering about related ideas and concepts (`smooth pursuit'),
sometimes focusing on specific themes, sometimes moving to completely new ideas (`saccades').
One last post before I close for the night. There's another project I have been working on called Atmospherics (http://www.andreapolli.com/atmospherics), it's the sonification of weather data. It's gotten me thinking about the origins of language and voice so I want to bring this up generally now and go into it more in future posts.
How did art begin? Looking at the historical record, aforementioned cave paintings, mud sculptures, jewelery, etc. exist as dateable artifacts, and one can look to the traditions of native cultures to see the importance of performance and ritual. Clearly performance (dance, music) is an age-old tradition. How did this temporal art form develop? Are there influences in the natural world that are imitated/enhanced by performing? A very simplistic and cliched example might be the rain dance, a performance that attempts to have a direct influence on the environment. Certainly the experience of the weather is much like the experience of a ritual performance (beginning, middle, and end--it can even be compared to a narrative with a narrative arc and climax) Was the human experience of the weather the inspiration for the first performances? Is this experience still affecting our culture today? (think Wagner's opera's and Shakespeare's dramas). If so, is it possible to look at the cultures of various regions and analyze their cultural products (language, music, dance) based on the climates of the regions? Are people from areas more prone to violent weather more apt to create music with greater variations in pitch, rhythms, etc. Does any of this relate to the other writings in this blog or is this a completely different project? Food for thought, that's all for now.
All the previous posts to this blog were written several months ago and have been added here as a background to the project. Today I attended a day-long workshop on artistic uses of physical computing held in an artist's loft building in Brooklyn. The students at the workshop were a mix of programmers, engineers, former dot-commers, visual artists (primarily video artists), and performance artists. What brought the group together was a desire to create works that respond to the environment (environment primarily consisting of humans in physical space, but also including other not directly human related aspects of the environment). The workshop was full and another workshop was to be opened for students not able to get a seat in the first workshop. There is clearly a strong movement of artists who want their work to respond to the environment.
This is not necessarily new, artist's throughout time have both responded to the natural world and hoped that the natural world/environment would respond to their art (think of cave paintings believed to be representative of the desire for a successful hunt). What is new is the artist's capability through the developing technology to specifically define how this interaction will occur.
It feels like at present, building interactive experiences that we are like toddlers learning to interact with the world. Through trial and error we are learning what kind of reaction is caused by certain kinds of actions. Except instead of exploring possibilites that exist, we are creating new trajectories.
History:
The experimental method and perception research
-Philosophy, Physiology, and Perception
-Fechner and the beginning of experimental psychology
-Contemporary moving out of the laboratory into the real world
How vision research influences art and media
-Test for colorblindness and pointillism/seurat
-Photography and the retinal image
-Marey, Muybridge, and Stroboscopic vision and film
-Television and Benham's Top
-Stereoscopic photography (Amy Ione)
-Synaesthesia
Theory:
-The physical construct of vision as related to consciousness and How our visual system shapes thought (Luciano Dan Fontoura Costa)
-How the visual relates to ‘reality’ or ‘truth’
-Vision and symbolism/metaphor
-Realism in art and how this relates to current trends in technological art (Iimura)
-Art as a model of the visual process
Chapters:
The Past of the Eye:
-The evolution of the human eye
-Other kinds of living eyes
-Insect vision, the compound eye
The Future of the Eye:
-The Artificial Retina
-Retinal scans
-Computer assisted sensory devices (Cochlear implants tongue sensors, translation of vision to sound)
-Panoramic vision (inspired by insect eyes)
-Infra-red and other spectrum viewing devices
-Remote viewing
-Genetic Enhancements/Vision: The eyeless gene (Drosophila Melangaster)
The Simulation of the Eye/Recreating Vision
-Persitence of vision
-Frame Rates
-Pixel resolutions
Constructing the Scene
The auditory scene
The visual scene
Macro/micro construction
Improvisation
Structure of social systems/complexity
Soundwalk art
Locative media art
Virtual Vision:
Virtual reality and vision
Vision and the screen (physical effects)
Augmented Reality
Stereoptics
Integrating vision with other senses
Active Vision:
Eye Movements
Gaze Monitoring
Interest and Emotion Sensitive Media (IES)
Driving
Video tracking and analysis in art works
Robotic/Computer Vision
Insect vision/the compound eye
Physical movement
Image recognition
Biometrics
Vision and Space:
The Fly’s Eye and architectural applications of visual tracking systems
Satellite Imagery
Augmented vision
Distributed Vision:
Vision and the network
Streaming video/audio
Conclusion:
Similar Books
The Body Electric: An Anatomy of the New Bionic Senses James Geary Rutgers University Press NJ
Architecture and Identity: Responses to Cultural and Technological Change Chris Abel 2000 Focal Press
The Computer in the Visual Arts Anne Spalter 1999 Addison Wesley
Abstracting Craft: The Practiced Digital Hand Malcolm McCullough 1996
Digital Mosaics Steven R. Holtzman 1998 Touchstone
See http://www.khm.uni-koeln.de/~an/imagery/
created as part of the diploma project of An Reich at the Academy of Media Arts Cologne under the supervision of Dr. Seigfried Zielinski and Phillipp Heidkamp
Intuitive Ocusonics was first performed as a part of Meme Me: Identity and the Replication Age at Artemisia Gallery, in Cache at Columbia College Chicago in conjunction with ISEA97, and as part of a performative lecture at Imagina98. For a shockwave simulation of Gape see: http://homepage.interaccess.com/~apolli/gape.htm
Lindberg, David C. Theories of Vision: From Al-Kindi to Kepler (Chicago: The University of Chicago Press, 1976).
"In music, this relates directly to performing actions that can manifest in a special state of alertness, with great speed, and in close communication with other performers. ...Finally, a state of performance consciousness is sometimes achieved, especially during improvisation, in which one can be surprised by one's own actions and choices, seemingly arising from a special part of the mind, separate from that normally associated with conscious determinations." Rosenboom, David Extended Musical Interface with the Human Nervous System Assessment and Prospectus Leonardo Monograph series 1990 Published by the International Society for the Arts, Sciences, and Technology
Palmer, P. 85 (from Convergence bib)
Josephson and Carpenter
Maurice Merleau-Ponty, Phenomenology of Perception, trans. C. Smith, Routledge & Kegan Paul, 1962, p. 136.
Chapter 7 of H. Dreyfus What Computers Still Can’t Do, M.I.T. Press, 1991.
Minding Vision Luciano da Fontoura Costa - Cybernetic Vision Research Group IFSC-USP, Oct 1996
Roger Penrose, mathematician: 'The Aesthetics of Science"
Arthur Robinson and Barbara Bartz Petchenik book "The Nature of Maps"
When things aren't as they seem:
Moving optical illusions (moon illusion, other examples)
Driving
Fog (low contrast) movement illusion
Motion Induced Blindness
Color Illusions
Benham's Top
Afterimages (american flag negative example, negative scene glasses Stuart Antis)
Sensory crossover/Synaesthesia (artists Kandinsky, Paul Hertz, Jack Ox, Steve Boyer)
The Forgiving Eye
Color palettes
Pixel resolutions
Filtering (Nancy Burson's artwork)
Frame rates/Stroboscopic vision
Retinal implants
Augmented reality
The CAVE and Virtual Reality (Char Davies)
Stereoptics
How the eye identifies:
Image recognition
Description of how exactly the eye/brain/perception works (as it is understood presently)
Perception of the moving image
Machine vision (2D and 3D)
"To those who wonder if we will be able to build a thinking machine, it should be reminded that we have not even succeeded in producing seeing machines allowing performance comparable to those to be found in nature... In fact, many of such creatures have indeed been shaped by vision, a much more sophisticated and complete way of perceiving a challenging world. ...It is even possible that great extent of the functional architecture of the brain may have been determined by the specific demands imposed by the processing of visual information." (Luciano da Fontoura Costa)
How the eye is tracked and individuals are identified:
Security cams and privacy issues (talk about Steve Mann's artwork)
Vision in techno/cyber-culture
The camera lens as symbolic of the eye
--Eye movements and eye tracking
--Using the eyes to communicate
The Language of the eyes
'Think ye by gazing on each other's eyes / To multiply your lovely selves?' (Shelley Prometheus unbound, act 3, scene 4)
1. Eye contact
'Imagine you walk into a crowded train. You see a remaining empty seat, so you go across and sit down. You get out your book, and settle into it. During the journey, you become aware of a feeling that someone is looking at you. You glance along the carriage and, sure enough, someone is looking at you. As soon as you make eye contact with the stranger, he looks away. To my mind, this phenomenon is rather striking, in that it is not immediately obvious how you would have known that someone was looking at you, if you were already engaged in another activity.' (Baron-Cohen, Simon page 97)
causes physical arousal
"When you notice that another's eyes are looking at you, your heart rate starts to soar, and this physiological arousal can be measured in the brain's electrical activity, deep in the brain stem" (Wada 1961; Nichols and Champness 1971)
Christopher Tyler's work showing in conventional portraiture the positioning of one of the subject's eyes centrally in the composition. He also suggests that aesthetic judgements are acted upon by unconscious factors (eye contact?)
Tyler, Christopher W. An Eye-Placement Principle in 500 Years of Portraits Smith-Kettlewell Eye Research Institute, San Francisco.
Eye contact occurs as a friendly signal and a signal for reconciliation and occurs between mother/child/lovers
2. Desire
Looking infers 'wants' or 'has goal'
"All seeing is heated. It must always involve force and desire and intent. Even when I think I am least interested, I am already on the prowl. It doesn't matter what I'm looking at--a watch, a shiny hook; 'just looking' is a lie. I am always looking out, looking for, even just looking around." (Elkins, page 21)
candy example
3. Gaze Monitoring
monitoring that two people are looking at the same thing--either follow or direct
'Gaze monitoring (Scaife and Bruner 1975; Butterworth 1991) is seen in infants from around 9 months and in all children by 14 mos. The infant turns in the same direction that another person is looking and then shows gaze alteration, checking back and forth a few times to make sure that they are both looking at the same thing. Directing visual attention to a shared focal object.' (Baron-Cohen, Simon page 48)
Retinal scans and iris dilation detection
Emotion and vision:
Effects of color on mood
The effect of the screen on the eye and vision
Strobe effects (Japanese cartoon that caused seizures example)
Images causing pain, headaches (migraines) and blindness
"There are three times as many nerve connections between the ear and the brain as between the eye and the brain" (Alfred A. Tomatis) "We learn more through the ear than through the eye."page 16 From Berendt, Joachim-Ernst The Third Ear: On Listening to the World Henry Holt and Co., NY 1985
Introduction
Contemporary society is vision-focused
The physical construct of vision is related to our construction of consciousness (Luciano Da Fontoura Costa and Phenomenology Merleau Ponty)
Eye vs. ear in culture outer/inner yin/yang
"For the Chinese the eye is the yang sense expressing the sun and masculinity and the ear is the yin sense embodying the moon and femininity. Early humans were ear-centered, patriarchal society is focuses on the eye. Eye is dominance. They eye of the eagle" (and for that matter, the eye on the top of the pyramid on the dollar--ap) (Berendt, Joachim-Ernst page 27)
Description of 'gaze' by a blind adult 'To look at something intensely. An equivalent would be to listen to something very hard.' (Baron-Cohen, Simon page 67)
According to Sufi tradition 'Seers are blind' Also Homer, Oedipus after he sees what he has done, etc.
Inner hearing/inner seeing (Berendt, Joachim-Ernst page 33)
'The eye is a peripheral sense because it is directed outwards and only comprehends the external person. The ear on the other hand is a central sense since the outer world enters the human soul through the ear, apprehending the concealed inner being. That can be demonstrated at any time in everyday experience since blind people are usually more inwardly sensitive, focused, and spiritual than those who can see.' (Diether Rudloff) (Berendt, Joachim-Ernst page 12)
We try to 'visualize' our senses
The eye as the mirror
The eye as the window (windows)
'Through the window of the eye the soul regards the world's beauty. For the eye endureth the soul the prison of human form. Without the eye that prison were its torment.' Leonardo Da Vinci (Berendt, Joachim-Ernst page 21)
Camera lenses
Symbolizing evil with the eye
"This seeing it aggressive: it distorts what it looks at, and turns a person into an object in order to let us stare at it without feeling ashamed. Here seeing is not only possessing...seeing is also controlling and objectifying and denigrating. In short, it is an act of violence and it creates pain." (Elkins, Page 27)
"Every photograph is a little sting, a small hurt inflicted on its subject, but even more: every glance hurts in some way by freezing and condensing what's seen into something that it is not." (Elkins, Page 29)
The belief in the evil eye exists in many cultures. Ancient Romans 'oculus fascinus' the fascinating eye; ancient greeks 'baskania'; hebrews 'ayn-hara'; syrians 'aina-bisa' Modern italians 'mal occhio' Naples belief is intense 'la jettatura' France 'mauvais oeil' Spain 'mal de ojo' Germany 'bose blick' Hollan 'booze blick' Poland 'zte oko' etc. southern India 'drishtidosham' (page 117 From Baron-Cohen, Simon Mindblindness: An Essay on Autism and Theory of Mind MIT Press, Cambridge 1995)
Mal-occhio, etc.
The 'male gaze'
Objectification of the subject
Summary of how some of this is becoming reality
In 1995, I traveled to Vilnius, Lithuania to work with a group of artists on an installation project in a 17th century Observatory on the campus of Vilnius University. One of the collaborators, a sculptor whose work involved transforming abandoned buildings, had fallen five stories from the roof of one of those buildings about one year before our meeting. Rasa had suffered a severe spinal cord injury and was unable to walk and control her movements fully. Meeting Rasa and talking with her was an eye-opening experience for me. Even though her movement and speech were extremely limited, she was able to communicate with everyone in a deeply personal way. Not only that, but she was able to communicate her ideas about her sculpture to the other artists and they were able to produce her artwork for her, even though she was unable to create detailed drawings. I was deeply affected by meeting Rasa and continue to be puzzled by one question: How was it possible that without detailed speech or movement one person could communicate so much?
The question was on my mind particularly that year because I had volunteered to be one of the first instructors at my college to teach a course using a videoconferencing system to students several hundred miles away. The course was structured to accommodate twenty-five students in a classroom with me in Chicago and ten students in a classroom in Springfield. The videoconferencing system in place was the state of the art. The system contained several remote controlled cameras including one that automatically followed the instructor's movements in the classroom. There were two large screens in which we could see the students in Springfield. From the main base in Chicago we could control the cameras in Springfield to focus on a specific student. I had several training sessions on the use of the equipment and techniques for effective videoconferencing. Everything was in place for the highest level of interaction, but I still found it incredibly difficult to communicate with the students in Springfield. Other instructors who volunteered for this pilot program also expressed difficulties. How was it possible that a state of the art system that allowed the transfer of both visual and aural information in very nearly real time operated by experienced teachers could feel so unnatural and limit communication so severely? What were technologically mediated communication systems lacking?
I began to study, from an artist's perspective, aspects of person-to-person communication and how these might be explored as ways to enhance remote communication. One year later, Gape, my first performance using eye tracking, was created. The system I developed used a simple software based eye tracking system that determined the position of light and dark pixels in a live video signal. As the performer, I would look at a grid of nine words. Through looking, I would trigger spoken words (digital audio samples of my own voice). The audience of Gape perceived a performer trying to 'speak' a coherent sentence. I used an eye tracking system and disabled my hands and body in this performance to comment upon the simultaneously enabling and disabling effect of technology in contemporary life.
In retrospect, I think this performance related to metaphorical connections in my mind between the computer screen and the retinal image. The screen, made of a flat grid of pixels, is like the retinal image, a flat projection of color and pattern. The retinal image itself lacks the depth and meaning of the real world, and it was my understanding at the time that it is only through the process of interpretation that an understanding of the world is formed. I thought, in a similar way, information stored on the computer has no real meaning until it is interpreted. Information must not only be interpreted by the computer program, but also interpreted by a human interacting with the machine. The Platonic version of vision in which the eye emits a virtual fire, a material substance that formed the visible image called simulacra, that allows vision to take place is a very active one. In Gape, the image of the eye, usually the receiver rather than the transmitter of an image, was received by the computer. The computer then took this very material information (the bits and bytes that make up the image) in real time and translated it into sound perceived by the ears of the viewer.
I continued to develop this eye tracking system to create sound and by 1999, free improvisational performance with the system I called Intutive Ocusonics became very important to me. At the same time I was performing and recording, I began looking at some of the research analyzing the process of improvisation in order to look for connections between intuition, improvisation, and human computer interaction.
David Rosenboom has written on the use of brain wave information in human-computer interactive musical systems. He describes an automatic effect in his monograph on non-traditional musical instrument interfaces:
"In an ongoing improvisation situation, one may intuit the likelihood of occurrence of events, possibly involving unconscious calculations of event probabilities and, in rapid-fire sequences, predict event occurrences along with their precise timings, while attempting to execute events synchronous with these predicted timings. One senses that a synchrony is about to occur and makes a decision to 'go for it' more rapidly than one can utilize slower, reasoning processes. "
Rosenboom posits that musical improvisation can be both a controlled and automatic process, the same way eye movements can be both voluntary and involuntary. My own work with eye movements also focused on connections between conscious and unconscious movements. For models, I have looked to activities that require complex movements to be performed extremely quickly, for example driving a car or participating in a musical improvisation.
The exploration of the process of musical improvisation led me to question how an auditory scene is constructed. Josephson and Carpenter suggest that the construction of the auditory scene is also an internal process. I discovered that this was not inconsistent with the construction of a visual scene. In the algorithmic model, there are four known stages to the visual perception of objects.28 The image-based stage, the surface-based stage, the object-based stage, and finally the category-based stage. The object-based stage is the stage in which the perceiver's internal knowledge informs how the subject is perceived, and the category-based stage combines the sequences of eye movements and their corresponding images on the retina combine to create the functional 'idea' of the visual scene. 29
I found so many connections between the temporal process of creating and listening to music and the process of seeing, that it seemed clear to me that an artist studying or working today has to consider visual media a temporal rather than static medium. About the same time I was coming to this realization, I began teaching digital media in the Film and Media Department at Hunter College. The film and video makers and scholars here have shown me the connections between the evolution of film and the ongoing evolution of digital media. The moving image, once created with the ëwetí process of photographic emulsion is now produced more frequently using the ëdryí process of the digital. Film-makers have begun to work with simulated virtual sets and even characters, and there has been increasing interest in immersive virtual reality systems for the presentation of narrative. In addition, gaming has grown as a formidable part of the entertainment industry and has begun to be examined as an art form. An understanding of the complex process of human visual perception is required to make a computerized simulation seem real. This has caused an explosion of research into all aspects of human vision.
By looking at these phenomena and developments in the technologies of robotic vision, biometrics, augmented reality systems, retinal implants, etc., this book attempts to provide a framework that ties together contemporary vision research with the artistic production of the interactive moving image. This book attempts to consolidate what is currently known about human visual perception and how vision itself is perceived and modeled. Written for artists working with digital media interested in future directions of the medium.
|