Show Summary Details

Page of

PRINTED FROM OXFORD HANDBOOKS ONLINE (www.oxfordhandbooks.com). © Oxford University Press, 2018. All Rights Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a title in Oxford Handbooks Online for personal use (for details see Privacy Policy and Legal Notice).

Subscriber: null; date: 18 August 2018

The Recording That Never Wanted to Be Heard and Other Stories of Sonification

Abstract and Keywords

This article uses the story of the phonautograph as a way of investigating the development of a cluster of practices called “sonification,” or the transformation of nonsonic data into audible sound. It begins with introducing the phonautograph and reasons as to why its study is relevant under the field of sound studies. It considers the possibility of listening to a phonautograph recording as marking an aesthetic and epistemic shift in the history of sound. It then reviews a range of practices of sonification, all of which move data and experience between the sonic and nonsonic registers. This study advances three nested methodological propositions. It argues for attending to the modularity of sensory technologies, modularity of the relations between senses, subjects, and technologies, modularity of the senses themselves. Finally, the article concludes by returning to the case of the phonoautograph in order to advance some speculative propositions regarding the present conjuncture in the history of sound.

Keywords: phonautograph, sonification, nonsonic data, audible sound, modularity

From the Postsonic to the Sonified

In the spring of 2008, the incunabula of sound recording briefly broke into headline news.1 A group called First Sounds had played a recording that dated back to 1860 (Rosen 2008; Maugh 2008). Anyone with an Internet connection could now hear “the world’s oldest recording.” This accomplishment raised a number of epistemological and historical questions. What does it mean to speak of a sound recording made in 1860? Thomas Edison’s phonograph, first demonstrated in 1877 and patented in 1878, is still widely considered to be the first device capable of recording sound and playing it back (Read and Welch 1976; Chanan 1994, 1995). However, First Sounds had succeeded in playing back a recording from a device that predates Edison’s: the phonautograph. Invented in 1857 by (p. 545) Édouard-Léon Scott de Martinville, the phonautograph was conceived as a “sound writer.” It channeled sound through a conic funnel to vibrate a small membrane, which in turn vibrated a stylus, thereby creating visible tracings on a recording surface—first a piece of smoked glass and later a sheet of paper rolled around a cylinder. Scott called the images it produced “phonautograms,” literally, “speech self-writings.” The mechanical similarities between Scott’s phonautograph and Thomas Edison’s cylinder phonograph are striking, a point that was not lost on nineteenth-century inventors. In the early 1870s Alexander Graham Bell had experimented with the phonautograph as a means of teaching deaf children. It was a modified version of the device, employing a human ear as a diaphragm that, according to Bell, “gave him the idea for the telephone.” Edison also cited the phonautograph as an antecedent to his machines (Gitelman 2006; Sterne 2003; Bell 1878; Edison 1888). Yet, for all the similarities between the phonograph and the phonautograph, there existed no technology to play back phonautograph recordings as sound, at least until sometime shortly before 2008 (Chanan 1994, 1995; Hankins and Silverman 1995; Lastra 2000; Read and Welch 1976; Sterne 2003).

In this chapter we use the story of the phonautograph as a way in and out of investigating the development of a cluster of practices called “sonification,” or the transformation of nonsonic data into audible sound. The story of the phonautograph is remarkable because it was conceived as a technology for visualization. Its recordings were never supposed to be heard, but now, thanks to the availability of tools for transforming the nonsonic into the audible, we are able to play back and listen to a recording that was intended to be seen, not heard. To call this technological process “playback” is not quite accurate since, as we will see, a great deal of reconstruction went into First Sounds’ efforts. Rather, we consider the possibility of listening to a phonautograph recording as marking an aesthetic and epistemic shift in the history of sound. This innovation is more than just a technological feat; it also speaks to a significant change in the ways in which we construct distinctions among the senses. After further introducing the phonautograph, we review a range of practices of sonification, all of which are instructive insofar as they move data and experience between the sonic and nonsonic registers. Our conclusion then returns to the case of the phonoautograph in order to advance some speculative propositions regarding the present conjuncture in the history of sound. The relationship to sensation central to sonification (and the assemblages of media and practices on which it depends) is most distinctively characterized by the ability to transform data destined for one sense into data destined for another. This is not a claim about synesthesia or, worse, a desire to restore a so-called balance of the senses in an idealized sensorium. Rather, we argue that this extreme plasticity lays bare the degree to which the senses themselves are articulated into different cultural, technological, and epistemic formations. The implication for sound studies is simple to grasp but difficult to handle. If we acknowledge the plasticity of sound, we must also acknowledge the limits of a commonsensical definition of the field’s object as “what can be heard” or “sound as an effect in the world.” (p. 546) Scholars in the humanities and social sciences still too often treat particular technologies or cultural forms as if they are predestined for or determined by a single sense. If we take the proposition of the plasticity of sound seriously, it is no longer possible to maintain such assumptions. Recent decades of sensory history, anthropology, and cultural studies have rendered banal the argument that the senses are constructed. However, as yet, sound scholars have only begun to reckon with the implications for the dissolution of our object of study as a given prior to our work of analysis.

Why anchor a chapter on sonification to a visualization technology like the phonautograph? Scott’s phonautograph is often accorded a place in the history of sound technologies, but it was neither a sound-reproduction technology nor a sonification technology. The phonautograph’s relationship to sound could best be described as postsonic. It produced an image after the occurrence of a sound—a record of a past sound event that it did not seek to reconstruct sonically. According to Scott, the phonautograph was never intended to play back its recordings. Scott even derided Edison’s phonograph on the grounds that it “merely reproduced sound—it was not a sound-writer.”2 The phonautograph was part of a boom in nineteenth-century devices that rendered visible otherwise invisible aspects of the natural world. Through a variety of technical feats, these technologies rendered the processes of nature—sound, electricity, biological processes and rhythms—as visual data that adhered to an orderly pattern. Scott and many of his contemporaries believed that to see sound was to better know it. The point of the phonautograph was to render sound available to careful visual examination, not to make recordings for playback (Hankins and Silverman 1995; Levin 1990). First Sounds’ intervention was to ignore the inventor’s intent and instead consciously and deliberately reconstruct phonautograph recordings as if they had been made for playback.

It is a cliché of technological history that inventors’ intentions have a way of not working out and that whatever purpose or intention is built into a technology can change over time. The usual story is one of invention, innovation, engagement with users, and reconstruction ad infinitum as technologies achieve moments of temporary closure, only to be opened up again and reconfigured from a new, unexpected angle (Bijker 1995; Jenkins 2006; Pinch and Bijker 1984; Pinch and Oudshoorn 2003). The story of the phonautograph, however, is a little different. A minor device in the history of sound recording, it never really left relative obscurity. Even its most ardent users generally moved on to other options as the nineteenth and twentieth centuries rolled on (Scripture 1906). The phonautograph appeared only as a curiosity in histories of sound recording (when it was mentioned at all) until it was resurrected by a few media historians at the turn of the twenty-first century (Hankins and Silverman 1995; Lastra 2000; Levin 1990; Sterne 2003). Its brief reemergence and short-lived posthumous fame signal a shift that occurred in sonic epistemology in the intervening years. While Scott and his contemporaries sought to transform sounds into images, today groups of scientists, artists, and technologists work to convert nonsonic phenomena so that they may be heard and therefore known as and through sound. In its time, the phonautograph helped construct sound as an (p. 547) effect that could be reproduced without reproducing its cause. First Sounds’ accomplishment of sonifying a phonautogram reveals the plasticity of sound, its ability to convey data across different sensory registers. In addition, their work reveals the degree to which data are fluid and are not necessarily tethered to any one sense.

In the spirit of this handbook’s handbookness, we would like to call attention to the usefulness of the concept of articulation for our argument. Drawn from the field of cultural studies, articulation theory might best be described as antiessentialism in action (Grossberg 1992; Slack 1996; Hall 1986). Jennifer Daryl Slack and J. Macgregor Wise define articulation as “the contingent connection of different elements that, when connected in a particular way, form a specific unity.” For them, elements can be “made of words, concepts, institutions, practices, and effects, as well as material things.” Articulation theory is meant to draw attention to “the movement and flows of relationships” (Slack and Wise 2006, 127). So far, one might note similarities to Latour’s actor-network theory (Latour 1996) or the social construction of technology tradition cited earlier. Inasmuch as all three approaches are constructivist, there is a natural affinity. However, articulation theory and the cultural studies tradition from which it draws are more particularly attentive to questions of power and, for a lack of a better term, the density or cogency of articulations. Articulations:

can and do change over time. But here, too, the speed and direction of change is contingent. Some articulations remain relatively tenacious; they are rather firmly forged and difficult to disarticulate.… Others might be more easily broken and thus subject to disarticulation and rearticulation. It all depends on the particulars of the nature of articulations at any particular historical moment. (Slack and Wise 2006, 128)

In this chapter we use articulation in two ways: We show how sonification articulates a range of practices that render data for the ear, and we explore the ideas and practices articulated to hearing within sonification. However, our larger historical point is that, at the present moment, the articulations between particular senses and kinds of sense data are incredibly weak. If this is the case, it suggests that sound scholarship must be ever more vigilant about that shifting border between the sonic and the nonsonic.

Our chapter therefore advances three nested methodological propositions. We argue for attending to the modularity of sensory technologies; for the modularity of the relations between senses, subjects, and technologies; and, ultimately, for the modularity of the senses themselves.

Parameters of Sonification

Alternately known as sonification and auditory display,3 the practice of turning data into sound as yet has no obvious epistemic center, and its overall impact on the arts (p. 548) and sciences is as yet unclear. Simply put, sonification is the use of nonspeech sound to convey information. This involves the translation or transposition of data from a given sensory mode into sound—a rearticulation of sorts. Techniques for analyzing or presenting data have tended toward the visual at least since the Renaissance (Ihde 2009). While there are early examples of attempts to use sound as a primary means of monitoring the world—Leonardo da Vinci is credited with inventing a form of sonar as early as 1490—sonification is a relatively recent approach to rendering information sensible and has been promoted variously as a corrective or a supplement to visual display.

In her chapter in this volume, Alexandra Supper elegantly renders the shape of current debates about the definition of sonification and the policing of the term’s borders among people who work in the field. For our chapter we have chosen a broad definition of sonification to highlight both the range of practices that might be enclosed in its big tent and the connections between sonification and other sonic practices. According to Gregory Kramer and Bruce Walker (2004), two pioneers of auditory display, sonification falls into three basic categories: alerts and notifications, auditory icons, and audification. Alerts are the simplest instruments in the sonification toolbox. Fire alarms, sirens, the beeping of a microwave, and so on generally signify one thing: There’s a fire; get out of the way, an emergency vehicle is trying to pass; your soup is now hot. They are neither able nor meant to communicate anything more subtle than a binary possibility: “The fire alarm signals that there is or is not a fire; not where or how hot it is” (Kramer and Walker 2004, 151).

Auditory icons—or “earcons”—are capable of more subtle types of communication. Associated mainly with computing (Robare and Forlizzi 2009; Gaver 1989; Blattner, Sumikawa, and Greenberg 1989), auditory icons might give feedback on a process—scrolling through a window or moving a folder to the trash, for example. In the late 1980s, William Gaver (1989) developed a sonic interface for Apple called SonicFinder. The interface was an ambitious attempt to sonify computing but was incorporated into mainstream operating systems only in a limited fashion. Not only did Gaver’s system alert the user that a process was occurring, it also employed a level of sonic nuance that could communicate subtle features of these operations. For example, selecting a file would trigger a “hitting” sound that would vary in pitch and timbre depending on its contents or size. Remnants of this approach remain buried in the user interfaces of common software applications; in many applications, brief blips and sound effects, for example, signal the arrival of an email or its successful departure. Microsoft has programmed a veritable cacophony of noises into its Office applications, which users can enable or disable as they please. We know of no writer who actually uses the many earcons available in MS Word, but they are there and available should you wish to be the first.

While alerts and auditory icons communicate symbolically, audification functions isomorphically. Through the modification or the transduction of other vibrational or wave phenomena, audification renders data more immediately meaningful to a listener. In the early 1960s seismologist Sheridan Speeth (1961) proposed that amplifying and speeding up recordings of seismic movements could (p. 549) allow listeners to differentiate between earthquakes and explosions caused by bombs. Sonocytology—the sonification of cells—operates similarly by amplifying the inaudible vibrations of cellular life. As Sophia Roosth (2009) notes, yeast cells vibrate at frequencies within audible range. The volume, however, is far below the threshold of human hearing. Using scanning probe microscopes to record the vibration of these cells, sonocytologists have managed to sonify yeast cells and listen in on experiments. These scientists have found that adding alcohol to the cells’ medium causes them to vibrate more rapidly, a phenomenon that, when sonified, results in higher-pitched sound, giving the impression that they are screaming.

Many recent developments in sonification have happened thanks to computers or, more specifically, digitization. Here we confront one of the defining features of “new” media as described by Lev Manovich (2001, 45-48): transcoding. Because digital files exist as both computer data and as aesthetic representations of the sensible world (in the form of an interface), a wide range of possibilities exists for both the manipulation of data and the interface itself. When an audio editor displays the waveform of a recording, it offers a simple and fairly straightforward kind of transcoding. Given audio data, it renders a picture that represents some aspects of the audio recording’s waveforms. More creative options are also possible; for instance, an application called Metasynth allows users to actually generate sounds from the data found in pictures (http://www.uisoftware.com/MetaSynth/index.php). When lasers map the curves of old phonographic cylinders or trace the vibrations of old phonautograph tracings, they are similarly performing a kind of transcoding. Pace Manovich, transcoding is not necessarily a digital process. The study of pulsars in radio astronomy, for example, has for a long time rendered the rotation of distant stars as sonic data because the regularity of their rotations can be better heard than seen. However, transcoding becomes considerably more common in the digital realm because software allows for greater ease and facility in the manipulation of data sets. The pulsar example is instructive because it reveals one of the central features of audified knowledge, a feature that works because of transcoding—let us call it “effective indexicality.”

Audification requires an assumption about the indexicality of the reproduction or manipulation of a phenomenon. The listener must believe that the sound produced is, for all intents and purposes, made of the same stuff as the object it is meant to represent. While a microwave’s beep is symbolic and arbitrary, audification is indexical in that patterns of sound are directly caused by patterns in the data. Here we borrow the distinction between indexes and symbols from Charles Sanders Peirce’s semiotic theory. Peirce defined a symbol as a sign that has an arbitrary relationship to its signifier, such as words in language. He defined an index as a sign with a causal relationship to its signifier—a weathervane, for example (Peirce 1955, 107; Turino 1999). In audification, patterns in data are made to “cause” particular sounds to occur, which in turn render those patterns sensible in ways that may be harder to discern visually. Although there is an arbitrary dimension to the relationship—someone must choose which sounds to associate with which data—the resulting relationship is meant to be experienced as one of cause and effect. (p. 550) Human hands also construct Peirce’s weathervane, and its shape as a rooster or an arrow, while arbitrary, does not affect its indexical relationship with the wind. Because the indexicality here is a deliberate effect of a human-made representational system, we call it “effective”—it is more important for the audifier to believe that the relationship between sound and object is indexical than it is for the relationship to actually be indexical. (Some forms of indexicality in Peirce are less “human made,” as in the pain caused by a touching a hot stove, though there, too, we would note that a certain level of belief in pain as an accurate index of danger is required.)

Whether it apprehends information through symbolic or indexical representation, proponents of audification argue that the ear is particularly good at certain things, superior even to the eye. The ear’s ability to perform monitoring tasks has been exploited for decades with devices like the Geiger counter, sonar, and heart monitors (Kramer 1994). We can listen, recognizing patterns or anomalies, without tiring or even necessarily devoting our full attention to the task at hand. The sonification of patterned data—stock market trends, network traffic, vital signs, and so on—capitalizes on the ear’s ability to detect anomalies that might “pop out” of a continuous stream of information (Kramer 1994; Hermann, Niehus, and Ritter 2003). The eye isn’t as proficient at picking out discontinuities while the ear is highly attuned to “wrong notes.” In contrast to hearing, vision is often described as a focused sense, capable of parsing only one thing at a time. The ear, however, can discern and parse multiple sounds simultaneously. Proponents of sonifications characterize hearing as a faculty that is more sensitive to direction, a useful strength in analyzing spatial data or receiving and acting upon directional cues. Sonification discourse also suggests that the ear is more sensitive to affect than the eye (Barrass and Kramer 1999; Polli 2005), and, as such, talk about and around sonification often partakes of a long-standing romantic ideology that is attached to hearing in many cultural fields. The idea that the ear and the sonic arts are somehow inherently closer to the seat of emotion and affect is strongly rooted in Western traditions that shape the way we think about music, speech, and other forms of auditory expression (Flinn 1992, 7, 9; Sterne 2003, 15). We do not need to believe in the truth of these assertions, however, to appreciate their effectiveness in shaping how practitioners relate to, make use of, and construct arguments about sonification.

Aesthetics of Sonification

Perhaps it is the romantic idea of sound’s affectivity that has led to growing interest in sonification in the art world. While it is true that in most cases the making audible of information is largely utilitarian, in many others the lines between scientific (p. 551) and artistic production are blurred. As Alexandra Supper notes (in chapter 10), this is a major point of concern among some advocates of sonification since the field is itself seeking legitimacy among scientists. This attempt at disciplinary respectability is complicated by the relative impermeability of the boundaries between art and science. As Supper shows, sonification researchers who are inclined toward interdisciplinary projects often have no choice but to align themselves with the terms, language, and expectations of one field or the other. For practitioners whose work might be more clearly aligned with the empirical, presenting a project in an artistic mode might undermine its credibility.

There are precedents for the hybridization of the empirical and the expressive. Georgina Born and Andrew Barry (2010) develop a genealogy of the emerging field they call “art-science.” They identify three “logics” operating in these fields: accountability, innovation, and ontology (105). First, the aestheticization of data often brings a work to the attention of a public that would not necessarily normally have access to it. Insofar as an artistic mode of presentation brings researchers’ work into a wider public, it is submitted to scrutiny, and researchers are forced to account for their findings. Second, the art-science pairing is often touted as producing innovations through synergy and collaboration. Third, artistic interpretations and presentations of research often work to call the very nature of scientific inquiry into question. Here we find art and science seemingly offering each other a chance at mutual legitimacy. Science stands to benefit from art insofar as the aesthetic putatively performs the function of making the empirical accountable (109). On the other hand, art practices gain epistemological credibility thanks to their supposedly creative contributions to knowledge (110).

Since its beginnings as a coherent field, this push-pull between art and science has played out in sonification discourse (Supper 2010). Despite its appeals to scientific rigor, sonification is an interpretive and aesthetic process. This is true for the actual sounds it produces, the reactions of listeners, the interfaces of sonification technologies used, and the relationships between sonic and nonsonic data in specific sonifications.

Projects by several individuals—and in many cases teams—present sonified data sets as works of art. Andrea Polli’s (2005) “Atmospherics/Weather Works” employs a fifteen-channel sound system to re-create significant storms in the New York/Long Island area. Polli and her team mapped several variables—atmospheric pressure, relative humidity, wind speed, and so on—to an assigned set of recordings that included vocals sounds and wind instruments, as well as environmental and insect sounds. As such, “Atmospherics” doesn’t simply present recordings of the storms; rather, it uses arbitrarily assigned sounds to create an affective representation of weather. Polli’s stated intentions make it clear that her work is meant to straddle the line between art and science:

As an artist sonifying atmospheric data, I am interested in the creation of new forms of data interpretation. As individuals and groups are faced with the need to interpret more and more large data sets, a language or series of languages for (p. 552) communicating this mass of data needs to evolve. In my artwork, I have tried to develop strategies for the interpretation of data through sound that has both narrative and emotional content because I believe that an emotional connection with data can increase the human understanding and appreciation of the forces at work behind the data. (Polli 2005, 33)

Other similar examples include “Life Music,” a collaboration between composer John Dunn and biologist Mary Anne Clark (1999) that sonifies DNA sequences by assigning pitches to amino acid sequences; Thierry Delatour’s “Molecular Music” (2000) makes audible the vibrational spectra of molecules; Navegar é preciso, viver não é preciso by composer Alberto de Campo and sociologist Christian Dayé (2006) sonically represents a variety of socioeconomic data using the time/space coordinates of Magellan’s 1519–1522 circumnavigation of the globe.

The practice of using data as raw material for art has a pedigree that reaches back at least as far as the conceptual movements of the 1960s. Sound artists, including John Cage, Alvin Lucier, and Charles Dodge, along with installation artists like Hans Haacke, Sol Lewitt, and Dan Graham, broke away from the prior conception of artists as individuals that realize works according to their own uniquely personal vision. One of Cage’s many contributions to the dismantling of the romantic notion of the artist as inspired genius was the work Atlas Eclipticalis, in which he superimposed music paper on top of star charts and plotted musical compositions as though they were constellations. In I Am Sitting in a Room Alvin Lucier sonified the acoustic characteristics of a room by recording his speech and then playing the recording back into the room, rerecording, playing it back, rerecording, and so on until his vocal signature was thoroughly eroded by successive generations of mediation.

Works like these brought natural and artificial systems to the fore by putting a process or phenomenon ahead of the artist’s ostensibly personal “vision.” What these works also implicitly proposed was that phenomena beyond the perception of human senses might somehow be represented or reified. Lev Manovich, writing about contemporary visual art that has followed a similar line, suggests:

If Romantic artists thought of certain phenomena and effects as un-represantable [sic], as something which goes beyond the limits of human senses and reason, data visualization artists aim at precisely the opposite: to map such phenomena into a representation whose scale is comparable to the scales of human perception and cognition. (Manovich 2002)

As we’ve seen, this move to render data sensible is at the core of sonification. In this sense artists and researchers not only are allied in seeking a common objective but often also work together and occasionally even take on both roles. These artist/researchers, who often publish in journals like Leonardo or Computer Music Journal, have tended to present their work as a fusion of art and science. The justification is that sonification can provide real insight into physical or informational processes, as well as serve to create or reinforce a sense of immediacy or relevance. The artistic presentation of sonified data is the verso of the phonautogram’s (p. 553) recto. Resolutely experimental in its approach and outcomes, the phonautograph occupied an ambiguous place between art and science. Perhaps this is a central feature, more generally, of the process of transcoding from one sense to another. Not quite synesthetic, not quite scientific in a classic sense, the conversion of data between the senses nevertheless presents itself as having a certain empirical weight and is conditioned by the aesthetic sensibilities of inventors, experimenters, and users.

Accessibility

Sonification discourse has predominantly been concerned with the interpretation and display of data, but it has also been widely touted as an important tool for creating accessibility for people who are blind. The benefits of sonic alerts, earcons, and audified information for those with limited or no vision are obvious, but there is one sonification technique that is arguably even more useful: human echolocation. Through rigorous training, some blind individuals have learned to navigate the world with remarkable aptitude. Daniel Kish, the most notable proponent of human echolocation, lost his sight in his infancy but has gradually learned how to negotiate the world acoustically (Kish 2009). By emitting vocal clicks, Kish manages to negotiate the world with a grace that is incomprehensible to sighted people. Even more astonishing, he is able and has taught others to ride bicycles. He is also able to sense and describe objects around him in detail; with training and practice echolocation is not only able to reveal the presence and placement of objects but can also help a listener to discern size, texture, and density.

In that echolocation is a technique that uses sound to convey or relay information, it is consistent with the most basic definition of sonification. However, unlike other forms, it does not operate in the symbolic realm, creating representations of data or phenomena. Rather, it is a technique for emplacement, for an immediate experience of a space. The sounds that the echolocator produces and then receives do not represent or tabulate the world; they present it in its immediate plenitude. In the same way that the reflection of light off an object constitutes the unmediated experience of seeing, the reflection of sound off a thing constitutes an unmediated experience of hearing. This is arguably true of a recent technological extension of human echolocation, FlashSonar (Kish 2009). Developed in part by Kish, the device emits clicks at a rate, volume, and endurance beyond the capacity of a human. While the sound that the FlashSonar employs has been arbitrarily chosen, the “image” of the world that the listener receives is not represented; it is immediate.

In all of these examples of sonification and the discussions it has precipitated, one might detect a slight corrective. The argument is that data have historically been largely represented by visual means. This should come as no surprise given the (p. 554) inherent visual bias of Western science (Ihde 2007, 2009). There is a long history in both scientific thought and the philosophy of science of privileging vision as the sense most likely to yield scientific knowledge. Vision has not only been favored epistemologically but has also been granted metaphorical dominance. The prevalence of light and sight metaphors have been with us at least since the ancient Greeks; Don Idhe and others have brought awareness to this visualist legacy in scientific thought (Peirce 1955; Foucault and Gordon 1980; Virilio 1989; Jay 1993; Hillis 1999; Turino 1999). To be more precise, this bias is not toward vision as such but a particular construction of sight as a sense that objectifies and separates subject from object. One result is that while there exists a long history of rendering sound as visual data—a history coterminous with the rise of modern acoustics and hearing science (Kittler 1999; Ferguson and Cabrera 2008), sonification has only recently gained traction as an epistemological innovation in itself.

Reanimation as Sonification

Here we return to the First Sounds’ work, for in a way their project inverts the old visualist bias. If sound could be transcoded into image, it stood to reason that image could be transcoded into sound. The phonautograph was designed to turn sound into image, not to play it back. However, First Sounds were able to reverse this process and thereby render the images on the phonautograms as sound. The group’s 2008 success in playing back one of Scott’s phonautograms was accomplished using technology inspired by new practices in the preservation of old phonographic cylinders. Because some cylinders are close to one hundred years old, laying a needle on them might potentially damage (if not destroy) them in the act of playback. Instead of laying a needle on the grooves of a wax cylinder, newer apparatus use a laser to scan the media; computer software then reconstructs the audio. In the case of the phonautogram, a sheet of paper blackened by the smoke of an oil lamp and then scratched with a stylus, the process is similar; a “virtual stylus” scans the document and converts the markings into digital information that can be rendered as sound (Hennessey and Giovannoni 2008).

Again, unlike phonographic cylinders, phonautograms were never supposed to produce sound: Scott’s phonautograms were meant to be “read.” What is perhaps most significant about First Sounds’ achievement is that they succeeded in contravening Scott’s intentions by reading these objects so that we can hear them. This step has been problematic since the process was invented: The patterns that phonautograms display do not have any apparent meaning to the untrained viewer. This opacity prevented early would-be users from finding much use for the objects. Bell, for example, had hoped to use the device to train deaf students to speak by teaching them to modulate their voices until the phonautogram looked like one from his own speech. While elegant, the plan was unsuccessful because the (p. 555) phonautograph did not provide the necessary visual detail (Bell 1878). This remained the case until it became possible for lasers to scan cylinders. Following this innovation, the members of First Sounds hypothesized that it might also be possible to extract sound from phonautograph recordings (though the idea of hearing phonautograms probably extends all the way back to their first public presentation). They retrieved Scott’s original patent deposits from France and then digitally encoded them at Berkeley Labs. Their first few attempts were largely unintelligible; because Scott’s phonautograph lacked a governor and had to be cranked by hand, the recording speed was erratic and had to be calibrated to be intelligible. First Sounds had the arduous task of manually reworking the speed of the recording so that it played at a consistent pitch and rate (http://firstsounds.org; Giovannoni to Sterne 2008). Sound machines always need a little help from their human auditors, but in this case the phonautograph required at least three kinds of assistance—from its inventor, from First Sounds, and from us, its listeners.

First Sounds’ first success was an 1860 recording of “Au Claire de la Lune.” The original phonautogram had notations that signaled which part of the line was made by which word. First Sounds had known what they were looking for all along; what these clues offered was a map. Phonautographic recording couldn’t take advantage of modern engineering practice: The machine was not capable of time stamping and did not use reference tones or pitches. We have already seen that, from Scott’s perspective, phonautograms, though sound recordings in the sense that they were sonic vibrations inscribed on paper, were not sound recordings in the sense we think of them today. Rather, they were something closer to sound inscriptions, fossils, or traces. Scott did not intend for them to be heard or understood in sonic terms; rather, they were meant to help observers understand sound through visualization. Perhaps the great irony of the device is that it failed as a sound writer but, with a good deal of help, ended up being involved in the historical reconstruction of sound from 1860.

Laughing with the Dead

The admittedly minor career of Scott’s phonautograph and its phonautograms illustrates the articulated nature of all sound technologies and sonic practices; that is to say, the phonogram is necessarily understood differently in varying formations. In the formations of nineteenth-century science—in an epistemological paradigm where seeing and knowing were closely coupled—the phonautograph was a part of a long line of machines that rendered physical processes visible. In this sense, it was almost incidental that the phonautograph was dedicated to the transduction of sound since it shared an epistemic relationship with graphical devices that traced a whole range of physical processes. In the formation of nineteenth-century sound reproduction, the phonautograph stands as part of an (p. 556) important shift toward machines that treated sound as a reproducible effect. In this alternative formation, the phonautograph’s ability to transduce sound is essential for its historical and cultural legibility. Today, neither of those formations holds, and the phonautograph’s means shifts again as a result. The possibility of playing back a phonautogram was unimportant to Scott; the device was a means of making sound more comprehensible in a paradigm where comprehension often meant visibility. However, the possibility of playing back a phonautograph is now important to people for a range of reasons. Sonification offers a utopian proposal similar to Scott’s hopes for phonautography—but in reverse. It promises to turn a world of information into sound. First Sounds’ work therefore lies somewhere in between the audification of nonsonic data and the digital transcoding and reconstruction of analog sonic data. In either scenario, Scott’s intentions for his machine have become irrelevant. A phonautogram’s indexicality makes it recognizable when sonified or, more accurately, almost recognizable if one is instructed in what to listen for.

There is an obvious novelty factor in hearing an 1860 recording, but there is more to it than that. By way of conclusion, let us step beyond that to consider some possible historical, aesthetic, and epistemological implications of First Sound’s sonification of phonautograms. As we have argued throughout, sonification is a particular articulation of the sonic and the nonsonic, one that points to an increasing vagueness of the borders around the audible world. Sound studies of the present moment must therefore wrestle in new ways with the boundaries of their objects. The field must let go of its axiomatic assumptions regarding the givenness of a particular domain called “sound,” a process called “hearing,” or a listening subject. This is not a call for a kind of everything-goes postmodernism but rather a reminder of the articulatedness of sensory technologies, sense data, and the senses themselves. In the rise of sonification, we note an increasingly forceful articulation of the senses as permeable and susceptible to transcoding. Thus, we end by wrestling with our own reactions to the phonautogram playback. Here are three possible interpretations:

1. The “we are now closer to Scott” interpretation. Most writers on the phonautograph appear to agree that there is an undeniable morphological similarity between Scott’s phonautograph and Edison’s phonograph. Nonetheless, there is an epistemic gap with Scott and Bell on one side and Edison on the other because of the latter’s privileging of inscription. If this is the case, one could conceivably read our present moment as coming around to Scott.

Thanks to the design of computer software, we live in an era of unprecedented visualization of sound. This is a result not only of sonification practices but also of the ways in which software engineers have chosen to display sonic data and to construct the sound-mixing and editing process through the design of computer interfaces. To be clear, this is a design decision and not a necessary consequence of digital technology. There are many more occasions to “look at sound” in a digital recording studio than in its analog predecessor. Those of us who have mixed audio with a mouse instead of a considerably more tactile mixing board know well the impulse to set values in terms of whole numbers and to draw straight curves in our (p. 557) edits not because they sound right but because they look right. First Sounds’ digital sonification of Scott’s phonautogram was similarly calibrated to the medium’s visual appearance.

2. Despite the prevalence of visualization, we are also living in an age of unprecedented sonification, which is potentially a reversal of Scott’s process. In some important ways, First Sounds’ enterprise demonstrates the degree to which contemporary thinking inverts Scott’s proposition—that the purpose of manipulating sound was to transduce it into visible data. Today, anything can be turned into sound, and in some cases it is quite useful to do so. Sound reproduction has extended into an ever-increasing number of spaces of everyday life (becoming mundane in the process), and more and more nonsonic phenomena have come to be comprehensible through sound. Friedrich Kittler has suggested that the sound-reproduction technologies of the late nineteenth century deprivileged the voice. Instead, it treated all sounds the same; it submitted them to a logic that dictates that “frequencies are frequencies” regardless of their source (Kittler 1999). Sonification takes this one step further by converting the frequencies of any given data set into an audible spectrum. There is no better example of this than our ability to now listen to sound recordings that were never supposed to be reconstituted as sound. With the phonautogram, as with the yeast cell and the pulsar, we can now listen to that which was never meant to be heard.

3. There is also a broader conclusion to be made here about the plasticity of data in digital schemes and the dissolution of old knowledge about the senses. In our age, the very idea of audition is so different from that of Scott’s era that one must wonder whether the sound represented by his phonautograms is really the same sound that we hear coming out of the speakers when we play it back today. In Scott’s time, sound was viewed as a substance, and different sounds were given different values. Bell and Edison probably still understood sound as having a certain “thisness” to it out in the world even though they also understood it as reproducible. For them, speech or music (or birdsong or wind) had a certain ontological coherence and essential character. In current use, sound-reproduction technology constructs sound as nothing more than a stratification of vibration by the body. Those vibrations within the audible range are considered sound; those outside the audible range are interpreted haptically or are not perceived at all. But even the unity of “sound” as a perceptual category is an illusion of language. We think of the ear as a transducer, as something that changes “sound” into vibrations and then into electrical and neural signals that are experienced as sound. While this integration exists at the perceptual level, it does not necessarily exist at the biological level. Cognitive scientists currently believe that different regions of the brain process different kinds of sounds. If there was a gap between Bell and Edison’s understanding of sound and Scott’s, one could posit another gap between our sonic world and the sonic world in which all three inventors operated.

If there is discontinuity on one stratum, there is continuity on another. In a visceral way, the work of First Sounds pushes the audible history of recording back seventeen years from 1877 to 1860. If Scott’s recording is intelligible to listeners (p. 558) as a recording, albeit a bad one, then the old Victorian obsession with the voices of the dead serenading the living sticks with us even now when the dead sing to us every day on classic rock radio stations. We might also tip our hats to Freud’s comments on the uncanny nature of recording (Freud 1989). When the “Au Claire de la Lune” recording was played back on BBC, the news reader couldn’t control her laughter as her script moved from technological marvel to obituary (Richards and Sherwin 2009). If she could not help laughing upon hearing a dead voice (barely) reanimated, perhaps it is because, in the age of sonification, we have the power to make anything laugh back.

References

Barrass, Stephen, and Gregory Kramer. “Using Sonification.” Multimedia Systems 7 (1999): 23–31.Find this resource:

    Bell, Alexander Graham. “The Telephone: A Lecture Entitled Researches in Electric Telephony by Professor Alexander Graham Bell Delivered before the Society of Telegraph Engineers.” October 31, 1877. New York: Society of Telegraph Engineers, 1878.Find this resource:

      Bijker, Wiebe. Of Bicycles, Bakelites, and Bulbs: Toward a Theory of Sociotechnical Change. Cambridge, Mass.: MIT Press, 1995.Find this resource:

        Blattner, Meera M., Denise A. Sumikawa, and Robert M. Greenberg. “Earcons and Icons: Their Structure and Common Design Principles.” Human-Computer Interaction 4 (1989): 11–44.Find this resource:

          Born, Georgina, and Andrew Barry. “Art-Science.” Journal of Cultural Economy 3(1) (2010): 103–19.Find this resource:

            Chanan, Michael. Musica Practica: The Social Practice of Western Music from Gregorian Chant to Postmodernism. New York: Verso, 1994.Find this resource:

              ———. Repeated Takes: A Short History of Recording and Its Effects on Music. London: Verso, 1995.Find this resource:

                (p. 559) De Campo, Alberto, and Christian Dayé. “Sounds Sequential: Sonification in the Social Sciences.” Interdisciplinary Science Reviews 31(4) (2006): 349–64.Find this resource:

                  Delatour, Thierry. “Molecular Music: The Acoustic Conversion of Molecular Vibrational Spectra.” Computer Music Journal 24(3) (2000): 48–68.Find this resource:

                    Dunn, John, and Mary Anne Clark. “ ‘Life Music’: The Sonification of Proteins.” Leonardo 32(1) (1999): 25–32.Find this resource:

                      Edison, Thomas Alva. “The Perfected Phonograph.” North American Review (1888): 647.Find this resource:

                        Ferguson, Sam, and Densil Cabrera. “Exploratory Sound Analysis: Sonifying Data about Sound.” Paper presented at the International Conference on Auditory Display, Paris, 2008.Find this resource:

                          Flinn, Caryl. Strains of Utopia: Gender, Nostalgia, and Hollywood Film Music. Princeton, N.J.: Princeton University Press, 1992.Find this resource:

                            Foucault, Michel, and Colin Gordon. Power/Knowledge: Selected Interviews and Other Writings, 1972–1977, 1st American ed. New York: Pantheon, 1980.Find this resource:

                              Freud, Sigmund. Civilization and Its Discontents. New York: Norton, 1989.Find this resource:

                                Gaver, William W. “The SonicFinder: An Interface That Uses Auditory Icons.” Human-Computer Interaction 4 (1989): 67–94.Find this resource:

                                  Gitelman, Lisa. Always Already New: Media, History, and the Data of Culture. Cambridge, Mass.: MIT Press, 2006.Find this resource:

                                    Grossberg, Lawrence. We Gotta Get Out of This Place: Popular Conservatism and Postmodern Culture. New York: Routledge, 1992.Find this resource:

                                      Hall, Stuart. “On Postmodernism and Articulation: An Interview with Stuart Hall.” Journal of Communication Inquiry 10(2) (1986): 45–60.Find this resource:

                                        Hankins, Thomas L., and Robert J. Silverman. Instruments and the Imagination. Princeton, N.J.: Princeton University Press, 1995.Find this resource:

                                          Hennessey, Meagan, and David Giovannoni. The World’s Oldest Sound Recordings Played for the First Time. 2008. http://www.firstsounds.org/press/032708/release_2008-0327.pdf (accessed February 22, 2011).

                                          Hermann, Thomas. “Taxonomy and Definitions for Sonification and Auditory Display.” Paper presented at the International Conference on Auditory Display, Paris, 2008.Find this resource:

                                            ———, Christian Niehus, and Helge Ritter. “Interactive Visualization and Sonification for Monitoring Complex Processes.” Paper presented at the International Conference on Auditory Display, Boston, 2003.Find this resource:

                                              Hillis, Ken. Digital Sensations: Space, Identity, and Embodiment in Virtual Reality. Minneapolis: University of Minnesota Press, 1999.Find this resource:

                                                Ihde, Don. “From da Vinci to CAD and Beyond.” Synthèse 168 (2009): 453–68.Find this resource:

                                                  ———. Listening and Voice. Albany: State University of New York Press, 2007.Find this resource:

                                                    Jay, Martin. Downcast Eyes: The Denigration of Vision in Twentieth-Century French Thought. Berkeley: University of California Press, 1993.Find this resource:

                                                      Jenkins, Henry. Convergence Culture: Where Old and New Media Collide. New York: New York University Press, 2006.Find this resource:

                                                        Kish, Daniel. “Seeing with Sound: What Is It Like to ‘See’ the World Using Sonar? Daniel Kish, Who Lost His Sight in Infancy, Reveals All.” New Scientist (April 11, 2009).Find this resource:

                                                          Kittler, Friedrich. Gramophone-Film-Typewriter. Trans. G. Winthrop-Young and M. Wutz. Stanford: Stanford University Press, 1999.Find this resource:

                                                            Kramer, Gregory. Auditory Display: Sonification, Audification, and Auditory Interfaces. Reading, Mass.: Addison-Wesley, 1994.Find this resource:

                                                              ———, and Bruce Walker. “Ecological Psychoacoustics and Auditory Displays: Hearing, Grouping, and Meaning Making.” In Ecological Psychoacoustics, ed. J. G. Neuhoff. San Diego: Elsevier Academic, 2004.Find this resource:

                                                                (p. 560) Lastra, James. Sound Technology and American Cinema: Perception, Representation, Modernity. New York: Columbia University Press, 2000.Find this resource:

                                                                  Latour, Bruno. Aramis, or the Love of Technology. Cambridge, Mass.: Harvard University Press, 1996.Find this resource:

                                                                    Levin, Thomas Y. “For the Record: Adorno on Music in the Age of Its Technological Reproducibility.” October 55 (Winter 1990): 23–48.Find this resource:

                                                                      Manovich, Lev. The Anti-Sublime Ideal in Data Art. 2002.Find this resource:

                                                                        ———. The Language of New Media. Cambridge, Mass.: MIT Press, 2001.Find this resource:

                                                                          Peirce, Charles S. Philosophical Writings of Peirce. New York: Dover, 1955.Find this resource:

                                                                            Pinch, Trevor, and Wiebe Bijker. “The Social Construction of Facts and Artefacts: Or How the Sociology of Science and the Sociology of Technology Might Benefit Each Other.” Social Studies of Science 14(3) (1984): 399–441.Find this resource:

                                                                              Pinch, Trevor, and Nelly Oudshoorn, eds. How Users Matter: The Co-Construction of Users and Technologies. Cambridge, Mass.: MIT Press, 2003.Find this resource:

                                                                                Polli, Andrea. “Atmospherics/Weather Works: A Spatialized Meteorological Data Sonification Project.” Leonardo 28(1) (2005): 31–36.Find this resource:

                                                                                  Read, Oliver, and Walter L. Welch. From Tinfoil to Stereo: Evolution of the Phonograph. New York: Sams, 1976.Find this resource:

                                                                                    Richards, Jonathan, and Adam Sherwin. “Why BBC Radio 4 Newsreader Charlotte Green Got the Giggles and Couldn’t Stop.” Times Online March 28, 2009).Find this resource:

                                                                                      Robare, Paul, and Jodi Forlizzi. “Sound in Computer: A Short History.” Interactions (January/February 2009): 62–65.Find this resource:

                                                                                        Roosth, Sophia. “Screaming Yeast: Sonocytology, Cytoplasmic Milieus, and Cellular Subjectivities.” Critical Inquiry (Winter 2009): 332–50.Find this resource:

                                                                                          Scripture, Edward Wheeler. Researches in Experimental Phonetics: The Study of Speech Curves. Washington, D.C.: Carnegie Institute of Washington, 1906.Find this resource:

                                                                                            Slack, Jennifer Daryl. “The Theory and Method of Articulation in Cultural Studies.” In Stuart Hall: Critical Dialogues, ed. D. Morley and K.-H. Chen. New York: Routledge, 1996.Find this resource:

                                                                                              ———, and J. Macgregor Wise. Culture + Technology: A Primer. New York: Lang, 2006.Find this resource:

                                                                                                Sterne, Jonathan. The Audible Past: Cultural Origins of Sound Reproduction. Durham, N.C.: Duke University Press, 2003.Find this resource:

                                                                                                  Turino, Thomas. “Signs of Imagination, Identity, and Experience: A Peircean Semiotics Theory for Music.” Ethnomusicology 43(2) (1999): 221–55.Find this resource:

                                                                                                    Virilio, Paul. War and Cinema: The Logistics of Perception. New York: Verso, 1989.Find this resource:

                                                                                                      Notes:

                                                                                                      (1) We would like to thank the editors, Douglas Kahn, and each other for helpful suggestions on the chapter.

                                                                                                      (2) Édouard-Léon Scott de Martinville, Le Problème de la parole s’écrivant elle-même: La France, l’Amérique (Paris, 1878), cited in Hankins and Silverman (1995, 137) and in Levin (1990). Levin cites Scott in support of his thesis that, just as early cinema was heralded as a transparent reproduction of images that would supersede national languages, so the prehistory of sound recording articulated an “analogous discourse of democratization and univocal, natural signs.” page 36

                                                                                                      (3) While the two terms have generally been used interchangeably, Thomas Hermann (2008) has recently suggested that “sonification” be used to denote the principle of rendering data as sound while “auditory display” should refer to the apparatus or technical system. Acknowledging this distinction, we use the term sonification as a catch-all.