From Biological Signals to Music:
The History of the Sonification of Fetal Heart Rate Variability [1]

The Body Electric

Carlo Matteucci recognized, during experiments on pigeon hearts in1843, that the hearts activity is based on electrical processes.

1882, the physiologist Augustus Waller derived the first ECG, on his dog Jimmy, by dipping its four paws into conductive silver chloride solution. He recorded the hearts currents for the first time in 1887, with the help of a capillary electrometer.

The technology was substantially improved in 1903 by Willem Einthoven, who developed the ECG into a useful diagnostic procedure in hospitals.

.

In 1906, Cremer [2] reported the first successful fetal electrocardiogram (EKG)

 
and in 1908 Hofbauer and Weiss [3] recorded the first phonocardiogram of the fetal heart sounds.  

Dr (later Sir) Thomas Lewis [4] corresponded with the Dutch physiologist Willem Einthoven from 1906, concerning Einthoven's invention of electrocardiography, and Lewis pioneered its use in clinical settings. Accordingly, Lewis is considered the "father of clinical cardiac electrophysiology"

In 1913, Lewis recorded fetal heart sounds together with the mother’s electrocardiogram at University College Hospital, London. By using a twin string carrier designed in that year by WH Apthorpe, he was able with only one Einthoven galvanometer to obtain these two simultaneous recordings. Lewis used the carbon microphone system which Einthoven had designed when he first recorded phonoardiograms in 1907.

Listening to the Body Electric 

For twenty-five centuries, Western knowledge has tried to look upon the world. It has failed to understand that the world is not for the beholding. It is for hearing. It is not legible, but audible. - Jacques Attali

Necessity is not always the mother of invention, sometimes inventions themselves can mother new inventions. To this end, art and technology have always been strange bedfellows.

Although the invention of the stethoscope  (invented in 1816 by René Laennec ) revealed the acoustic complexity of a hitherto silent world, it wasn’t until the invention of  the microphone, the loudspeaker and later the wireless valve that the amplification of sounds became possible and  this new world became generally accessible.

On October 9, 1876, Alexander Graham Bell and Thomas A. Watson talked by telephone to each other over a two-mile wire stretched between Cambridge and Boston. It was the first wire conversation ever held. 

1876 Bell Centennial Telephone

Amplification of sounds was made possible by the wireless valve. Major G. 0. Squier  of the United States Army constructed a heart transmitter in 1921 which made the heart sounds audible in a large room and, in addition, he transmitted them by wireless. Abbott of Purdue University devised a telephone transmitter in 1923, tuned so that the adult heart sounds could be heard on a loud-speaker in a large room.

Human brainwaves (EEG) were first measured in 1924 by Hans Berger [6]. His results were verified by Adrian and Matthews [7] in 1934 who also attempted to listen to the brainwave signals via an amplified speaker. 

The Electrical Amplification of Fetal Heart Sounds  [8] 

It was not long after the wireless valve was used to amplify the adult heart sounds that an apparatus was devised to magnify the sound of the fetal heart. Falls and Rockwood  in the United States seem to have been the first to describe a makeshift amplifier in 1923 which enabled the fetal heart to be readily counted several feet from the loudspeaker.

Hyman developed a machine in 1930,  called a fetal phonocardiograph, that gave tracings as photographic records of the fetal heart sounds after they had been converted into electro-magnetic waves by radio-amplification. Hyman found some gross irregularities of the fetal heart rhythm (fetal heart rate variability). 

Pommerenke and Bishop of the Department of Radiology of the University of Rochester described their amplifier in 1938 which obtained sufficient amplification of the fetal heart to make gramophone records.

De Costa  described a photostethoscope in 1938 which gave good amplication of fetal heart sounds and which carried a neon lamp so that the sounds were translated into flickers of light.

In 1960, Dr. Lee Salk [9] published the results of research, indicating that the sound of a mother's heartbeat has a calming effect on a newborn infant.

In a later study of 287 mothers, he found that both right-handed and left-handed women have a strong tendency to cradle their infants near their hearts. Salk theorized that mothers who hold their children near their hearts provide an auditory link that quiets the infants and enhances their growth.

Dr. Salk tested his theory by broadcasting recordings of a normal heartbeat in a nursery. Babies responded by becoming more tranquil than those in a quiet environment. Prolonged exposure during the first four days of life resulted in increased weight gain, his studies showed. By contrast, babies exposed to the sound of a racing heartbeat appeared agitated.

"From the most primitive tribal drumbeats to the symphonies of Mozart and Beethoven," he wrote in a report to the World Federation of Mental Health, "there is a startling similarity to the rhythm of the human heart."

The sound of a healthy heart:

The Sonification of Bioelectric Signals

The use of electrical signals emanating from nerve and muscle (bioelectric signals) to create music came into being in the late 1960's. 

The first instance of the intentional use of bioelectric signals to generate music did not occur until 1965, when Alvin Lucier [10], who had begun working with physicist Edmond Dewan, composed a piece of music using brainwaves as the sole generative source. In that piece, EEG electrodes attached to the performer's scalp detect bursts of alpha waves generated when the performer achieves a meditative, non-visual brain state. These alpha waves are amplified and the resulting electrical signal is used to vibrate percussion instruments distributed around the performance space.

In 1966 10 New York artists worked with 30 engineers and scientists from the world renowned Bell Telephone Laboratories to create groundbreaking performances that incorporated new technology. 9 Evenings: Theatre and Engineering was to be the first event in a series of projects that would become known as E.A.T. or Experiments in Art and Technology.

For one of these performances, Grass Field, Alex Hay [11] wanted to pick up body sounds: brain waves, muscle activity and eye movements. Pete Kaminsky, Fred Waldhauer and Cecil Coker built a battery-driven differential amplifier which had a peak gain of 80 db at low frequencies from 112 Hz to 10 Hz. The whole unit, batteries and all fit into a 1" x 3" x 5" box, no mean feat to do this in 1966. The signal from the differential amplifier was fed into a voltage-controlled oscillator, then to a transmitter, which sent the sound to the speakers. Electrodes were placed on Hay's head (EEG) and chest (ECG) and all the equipment was attached to a plastic plate fastened on Hay's back. These body sounds were heard through the speakers as Hay carefully laid out 64 numbered pieces of cloth. Here Hay is sitting in front of a television camera and the image of his face is projected on the screen behind him as Robert Rauschenberg picks up numbered cloths.

Alex Hay. Grass Field (1966):

Rosenboom developed an environmental demonstration-participation-performance event entitled Ecology of the Skin in 1970-1971. It involved biofeedback monitoring of brainwaves and heart signals from performers and audience members and their translation into a musical texture, along with synchronous electronic stimulation of visual phosphenes (colored patterns often seen with eyes closed) at cerebral light-show viewing stations for the audience. The electronic setup for this work included the capability of adjusting the degree of brainwave control over sound for each of 10 participants according to a simple statistical measure, the amount of time spent per minute producing alpha waves.

Another early experimenter was Manfred Eaton [15], who carried out experiments in music and bioelectric phenomena at the ORCUS Research Center in Kansas City during the 1960s and early 1970s. Eaton described extensive explorations in applying various bioelectrically derived signals, including brain (EEG) and cardiac (ECG) signals, to artistic projects in order to generate complex patterns for music, kinetic arts and television. 

In February of 1971, French composer Pierre Henry [16] gave a concert at the Museum of Modern Art in Paris consisting of an improvised performance of electronic music based on the live manipulation of sound patterns modulated by his own brain wave activity.  The device that made this possible was the “Corticalart,” invented by Roger Lafosse, a researcher and musician who in 1965 founded the Sigma festival of contemporary stage and visual art in the French city of Bordeaux.

 “Attached to the musician’s head, a system of electrodes, comparable to those used in the electroencephalogram, allows the detection of three kinds of electrical signals which convey the characteristic activity of certain zones of the cerebral cortex: alpha waves (states of relaxation, inattention, repose), beta waves (states of alertness, attention, activity, reaction), and “artifacts” caused by the movement of the eyeball.” 

From the album “Mise en musique du Corticalart de Roger Lafosse“ (1971):

Nam June Paik’s [17] video, A Tribute to John Cage (1973), is Paik's homage to avant-garde composer John Cage, a major figure in contemporary art and music.

In 1981, the composer/artist/architect Christopher Janney [18] began researching heartbeat monitor systems and modified a wireless telemetry system, equipping it with a custom audio filter which  isolated the sound of the heart’s electrical impulses to the brain and its surrounding muscles.

In 1982, Janney collaborated with choreographer/dancer Sara Rudner and developed  “Heartbeat”, a performance utilizing the customized heart monitor, with the focus on exploring the heart as both a machine for pumping blood and the “seat of the soul.” The result was first performed in 1983 at The Institute of Contemporary Art in Boston.

The dance is a solo piece, with choreographic structure within which improvisation is taken. The dancer wears a wireless device that amplifies and sonifies the natural electrical impulses that stimulate the heart to beat. This forms the basis of the musical score, which is then overlaid with sounds of medical text, jazz scat, and the adagio movement of Samuel Barber’s String Quartet.  

In 2001, Erich Berger [25] created A Sophisticated Soirée, a temporary installation space in which sound and visuals are controlled by the heartbeat of visitors, at the Ars Electronica Festival, Linz (A). 

The 64 participants where each fitted with two disposable stick-on electrodes, which register the electrical signal of their heartbeat and send this, via a wireless transmitter unit, to a receiver station. The signals were used to trigger various different musical and optical processes directed by computer programs. 

Erich Berger: A Sophisticated Soirée:

October 2006 was the premiere of Berger’s Heart Chamber Orchestra in Trondheim Norway. A sensor network consists of 12 individual sensors; each one is fitted onto the body of a musician. A computer receives the heartbeat data. Software then analyzes the data and generates, via different algorithms, the real-time musical score for the musicians, the electronic sounds and the computer graphic visualization.

Erich Berger: Heart Chamber Orchestra:



<  

By practicing a patient-specific breathing technique, under the guidance of a therapist, it is possible to bring the autonomic nervous system into a condition, such that it can regulate optimally. 

With the HeartMusic software program, these recorded data are then converted, in real-time, into music.

In real time, the heart’s melody reacts to all therapeutic steps. This patient-specific, breath-dependent oscillation of heart rate represents the wave which, when broken down into its individual time and frequency components and further analysed, forms the basis of the heart’s music to be heard.

If, under guidance of the therapist, a sufficient progress is recognized, the music of the heart is recorded for the patient during the exercises and handed out as an audio CD or mp3 file. This own HeartMusic is heard daily by the patient and the therapeutic process is thus resumed up to the following HeartMusic Therapy session. The clearly defined psycho-physiological, therapeutic effects of listening to music are well-known. This effect is greatly enhanced due to the unique source of the music. 

HeartMusic Therapy: Healthy HeartMusic:



 

HeartMusic Therapy: Chronic illness:

[2] Cremer, M.: Über die direkte Ableitung der Aktionsstroeme des menschlichen Herzens vom Oesophagus und über das Elektrokardiogramm des Fetus. Münch. Med. Wschr. 53 (1906) 811

[3] Hofbauer, J., O. Weiss: Photographische Registrierung der foetalen Herztöne. Zbl. Gynaek. 32 (1908) 429. Gynecol. 32 (1908) 429

[4] Hollman, A., Journal of the Royal Society of Medicine, Volume 82, November 1989, 694.

[7] Berger H., “Uber das elektrenkephalogramm des menschen”, Arch. f.Psychiat, vol. 87, pp.527-570, 1929. 

[9] Salk, L. (1960). The effects of the normal heartbeat sound on the behavior of newborn infant: implications for mental health. World Mental Health, 12, 1-8.

[11]«Grass Field», Performance  presented as part of 9 Evenings: Theatre and Engineering, The 69th Regiment Armory, New York, N.Y., United States, October 13-22, 1966. http://www.fondation-langlois.org/html/e/page.php?NumPage=662

[20] The Heartsongs project:  http://reylab.bidmc.harvard.edu/heartsongs/