Auditory Icons, Earcons, and Speech

Can Auditory Icons be Used More Effectively in Hearing Aids?

Background

One of the most successful improvements to the user-electronics interface made since the early 1970s has been the inclusion of icons (eye-cons) – graphic symbols that visually represent information in the electronics display (photos, mail, text, calendar, etc.), whether it be a personal computer, smart phone, or any of the numerous electronic gadgets on the market today. Icons have replaced much information displayed previously as text. Icons are able to present a great deal of information concisely, quickly, and with less visual exposure.

However, overly dense visual displays can lead to a “cognitive overload” that adversely affects a user’s decision-making performance. As a result, alternate channels of communication have arisen – and one in particular – that of using auditory cues/signals to convey additional information, is the topic of this post.

Although auditory cues are used with multiple electronic devices, the focus here will be related to how auditory cues are used in hearing aids, aside from general amplification implications. Many current hearing aids make use of auditory cues to provide information, primarily about environmental settings, loudness level, and battery condition. Audition is an obvious choice to provide additional information because this is the medium impacted and enhanced by hearing aids. Hearing aids do not have a visual screen associated with them that would allow icon use (other than those integrated with smart phone apps, and perhaps certain remote controls).

So, the purpose of this post is to differentiate the two primary auditory cues that are used with hearing aids, and to offer some thoughts as to how they might be used to help make hearing aids “more useful” to the wearer.

Hearing Aid Auditory Cues

Much of the work related to auditory cues has evolved from activities related to the personal computer. Different operating systems are recognized upon startup by their distinctive sounds; incoming messages have unique audio impressions, errors in completing an activity often provide a tone and/or visual impression (icon) telling you that you must do something additional, etc.

An Icon is distinct from the Auditory Icon and Earcon. The icon is a visual symbol display that simplifies the presentation of a lot of information in a concise and easily-recognized format. The visual system’s capability to process several dimensions such as shape, color, etc. in parallel, allows a variety of information to be encoded into a visual icon.

The same can be said about the auditory system based on its ability to process pitch, amplitude, quality, etc.), but with the information being presented via sound. In both cases, using a universally-recognized approach to denote information can transcend cultural and linguistic barriers. In the case of the icon, it is visual; for the auditory icon and earcon, the medium is auditory (Figure 1).

Auditory Icon or Earcon? Which is it, or are they the same?

Auditory Icon

An auditory icon is a brief sound that is used to represent a specific event, object, function, or action. The sound takes advantage of the user’s prior knowledge and natural auditory associations between sounds and their results. In a sense, auditory icons are essentially caricatures of naturally occurring sounds. Natural sounds are preferred because they are related to events in a principled, systematic way (described by physics), and people learn this mapping from early childhood in their interactions with the world.

In a sense, the sounds are meant to be the auditory equivalent of visual icons that are widely used in personal computing (where they represent objects or processes through graphical symbols), but in this case, represented by a specific, commonly-recognized sound related to the activity.

Auditory icons refer to recorded everyday sounds that are used for the purpose they represent, either directly or indirectly. Direct relations use the sound made by the target event, whereas indirect relations substitute a surrogate for the target, and require an additional learning process to develop the relationship between the sound and a specific event. It is preferable to avoid indirect relationship sounds.

Direct relationships can be characterized as follows: a telephone ring can be used to represent the telephone, a car horn to represent a car; a train toot for a train, etc. The directness or auditory similarity between the auditory icon and the actual object can vary considerably. But, as long as a sound evokes the associated sound of an object or action, it is classified as an auditory icon. The auditory icon need not be a realistic representation of the object/function it portrays, but should capture the essential features, in which case it would be identified as a representational auditory icon.

Earcon

Earcons are structured sounds, defined as a brief, nonverbal, distinctive audio sound/message to represent a specific event or convey other information and feedback to the user. Some examples of earcons (sounds) associated with personal computers are:

• The arrival of a mail message
• The system going off line in a few minutes
• The indication of a misspelled word in user input
• A sound equivalent of a file name or available tool
• A reminder that a file in temporary workspace has not been written to disk
• An out-of-bounds variable in a structured editor

Earcons differ from Auditory Icons in that earcons are generally synthesized tones or sound patterns, and have no direct relationship to the event. A learning process is involved for the indirect sound to eventually have a specific meaning. Although frequently associated with computer interfaces, earcons (not identified as such at the time) have a much longer history, such as:

• The alert signal from the Emergency Broadcast System
• The signature three-tone melody used by NBC in radio and television broadcasts
• Audio alarms and signals of various types

The earcon name has been in use since 1985 and is a pun on the more familiar term icon in computer interfaces and in many symbols used over the years, which are visual, such as “No Smoking” signs, etc. that merely display a smoking cigarette in a circle and line through it. There is some confusion as to who coined the term “earcon,” but it may have been inspired by D.A. Sumikawa as the auditory equivalent to icons in a 1985 article.

Earcons are used where there is no clear iconic representation, but can yield an effective sonification. Earcons are abstract, synthetic, and mostly musical tones or sound patterns that can be used in structured combinations. Earcons are non-verbal audio messages that are composed of “motives,” which are short, rhythmic sequences of pitches and variable intensity, quality, register, and dynamics. Because most earcons have solely symbolic mappings between the information they represent, the associations still require considerable learning by the user. They are used to add context, helping the user maintain awareness.

Speech – The Ultimate Auditory Icon?

The most obvious presentation of an object is conveyed by simple speech. Messages presented using voice seem to be assimilated with less effort than the same messages presented through visual media. If kept to a single one-syllable word, it might be considered the ultimate auditory icon because its intention is immediately recognized and understood. It seems no accident that voice became the first to be explored as a means of communication in the incorporation of sound in user interfaces.

In most ways, a voice command is managed in the same way as an auditory icon or an earcon. Attention must be paid to the frequency response, level, spectral content, etc. But, it does require something additional – the ability to understand the word, and especially if in the presence of other amplified speech. Hence, the need for a single syllable word presented at an appropriate time, in isolation.

Auditory Icon or Earcon?

Hearing aids today use a combination of auditory icons and earcons.

Earcons include the tones, chimes, bells, etc. that indicate the environmental program selection, or the low battery indicator. These sounds do not provide any direct auditory link between the signal and the event. The link has to be learned and is not a natural immediate understanding of the sound intention.

An important concern about the use of earcons to provide hearing aid information is that they must be easy to comprehend and remember to be useful. An auditory icon would be a telephone ring to identify the telephone listening program (rather than 1, 2, 3, or 4 beeps).

Auditory Icons/Earcons for Hearing Aids to Become More Usable

Are there uses of auditory icons and earcons that will help the hearing aid become a more usable device? Something that will allow wearers to use their instruments better/more efficiently? Following are some initial thoughts, for better or worse (most readers will be able to imagine other possibilities).

• Low battery
  • Use of a three-note intensity and pitch pattern decrease to metaphorically represent the need for a battery exchange. Many current hearing aids use a short series of pulsing tones or other sound to alert the wearer that the cell is in need of replacement. The question remains, however, does this represent something completely different than merely changing from one response setting to another, especially if/when the same tones are used? How does the listener make this distinction?
  • Low battery could be identified with something other than tones. The current use of tones occurs for almost all functions – volume changes (if available), settings, and battery replacement. I suspect this is confusing for many individuals, especially those advanced in age, but also for others who might have cognitive issues. Perhaps a single pitch earcon that decreases in volume could be used as an audio cue to indicate that the hearing aid battery needs replacement. The single, decreasing volume pitch becomes a metaphor for the application it depicts.
• Setting Changes.
  • Could auditory icons be used to alert the user to amplified changes taking place in a way that has meaning? Most people I know have a difficult time remembering what 1, 2, 3, or 4 beeps represent, and made even more difficult when they can’t hear any noticeable differences between the settings. As a result, many do not change settings. Don’t believe me, ask your patients. Substituting a telephone ring, noise burst, or other directly meaningful sounds would allow the user to know immediately which setting they were on. And if representative sounds are not available, what about the novel approach of a voice saying “home,” “noise,” “phone,” “group,” “song,” etc.? The words should be recognizably different. It might even be good to repeat the word, “noise, noise.” I think I am safe in saying that this would be more meaningful to many than hearing 1, 2, 3, or 4 beeps or tones. And, what if they don’t hear all the tones? What are they then listening to?
• Reminder.
  • Might it be a good idea to periodically use the voice word to let them know what setting they are on – perhaps shortly after they make the selection and then following some duty cycle?
• Malfunction.
  • The use of an earcon that represents a crash could alert the user that something is wrong with the hearing aid, as discovered during its initiation startup.

 

References:

1. Buxton, W., Baecker, R., & Arnott, J. (1985). A holistic approach to user interface design. Unpublished manuscript
2. Sumikawa, D.A., 1985. Guidelines for the integration of audio cues into computer user interfaces, Technical Report UCRL-53656 ON: DE85016506, Lawrence Livermore National Lab, CA (USA)
3. Dingler, T., Lindsay, J., Walker, B.N. Learnability of sound cues for environmental features: auditory icons, earcons, spearcons, and speech, Proceedings of the 14^th International Conference on Auditory Display, Paris, France, June 24-27, 2008
4. Elliott, F. R. (1937). Eye versus ear in moulding opinion. Public Opinion Quarter&, I, 83- 86
5. Sticht, T. G. (1969). Learning by listening in relation to aptitude, reading, and controlled speech (Tech. Rep. No. 69-23). Alexandria, VA: Human Resources Research Organization

---

Wayne Staab, PhD, is an internationally recognized authority in hearing aids. As President of Dr. Wayne J. Staab and Associates, he is engaged in consulting, research, development, manufacturing, education, and marketing projects related to hearing. His professional career has included University teaching, hearing clinic work, hearing aid company management and sales, and extensive work with engineering in developing and bringing new technology and products to the discipline of hearing. This varied background allows him to couple manufacturing and business with the science of acoustics to bring innovative developments and insights to our discipline. Dr. Staab has authored numerous books, chapters, and articles related to hearing aids and their fitting, and is an internationally-requested presenter. He is a past President and past Executive Director of the American Auditory Society and a retired Fellow of the International Collegium of Rehabilitative Audiology. 

 

_{**this piece has been updated for clarity. It originally published on September 22, 2014}