Inside ReSound Sensia: Automatic AI and Smarter Hearing in Noise

The week before the launch, select members of the media were invited to GN headquarters in Denmark to learn about the improvements incorporated into the new ReSound Sensia hearing aid and the corresponding Beltone model Illuminate. In a series of sessions, we got all the details, then experienced an excellent demo of Sensea’s hearing and noise features while listening through a mannequin. GN stressed the importance of their new system called AcousticIQ for identifying scenes and choosing the optimum hearing aid setting, including the DNN, which in the Vivia is selectable only manually. Automating the DNN functionality was an important goal for GN so that the hearing aid delivers the optimum user experience for everyone instead instead of relying on people remembering to try the ‘Hear in Noise’ program. But there’s much more to it than that. The next day I spent time with Laurel Christiansen, Chief Audiology Officer, and Brian Dom Pedersen, CFO, to get the comprehensive picture of the Sensia’s audiological benefits. Let’s hear from each in turn. Their explanations are quite complementary, and together they give a complete picture of the benefits people with hearing loss can expect. So thanks for joining me, Laurel. I’m sure most people here know who you are, and we’ve talked a couple of times before, but I’ll just introduce you as the Chief Audiology Officer for GN. Anything else people should know about you?

I don’t think so. I’ve been here for about almost 24 years now as the head of audiology for GN, and it’s always great to talk to you. Well, thank you.

So the Sensia is a really intriguing device to me. Partly because I think at this stage I thought that your classical signal processing methods of improving speech in noise had pretty much been tapped out. And so the next improvements were coming from DNNs. But you’ve still managed to do some things that improve it over the Vivia I’d like to explore that. So you’ve put a lot of emphasis on the scene classifier. So tell me what’s different about the Scene Classifier in this one.

I guess the thing I would say, number one, is that it’s very accurate. If you go into a restaurant, the hearing aid is going to know you’re in a restaurant, you want to hear speech. It’s going to put you in the right settings, you know, for the best hearing in that situation. We don’t classify music, which is really what complicates the situation. And it knows, you know, where people are and it can turn on the right features at the right time for that patient. And, you know, one thing is really important about that. Our own data shows that people change to the second program, only about 10% of people will change to the second program.

Meaning the ‘Speech in Noise’ program.

Yeah, the noise program that we have today, not a lot of people will take advantage of it. They want everything to happen automatically. But automatic is not easy. You have to have the right classification and you have to engage the right features. And so we have this new acoustic in ours, that’s what it’s called, and it’s both the classification and the engagement of the features, and it’s extremely accurate.

Okay, and it’s interesting because I always thought the Vivia classifier was really good. Rarely would I go out of the general mode, right? Of course, in the Vivia, you have to manually engage the DNN, and that would be the one case. But normally, it seemed to work well. What situations— leave the DNN out of it, right? But what situations would I find improvement in Sensia versus Vivia?

So what we’ve changed in Sensia, the environmental classifier is essentially the same. It’s classifying the same environments, but we’re actually engaging the features earlier now for hearing in noise. So we actually lowered some of the thresholds and we’re engaging the, you know, the what we call Better Ear Mode where you go into asymmetric directionality faster than we were before. And then we’re putting you into the deep neural network and binaural directionality a little bit more than you would have been in Vivia.

So let’s go through the steps, right? Because you’re engaging different features in stages. So I’m in a quiet environment. What’s happening?

So in a quiet environment, you’re omni in both ears. We want to emphasize spatial hearing awareness of everything around. So in a quiet environment, you would always be in two omnidirectional, and we call that Spatial Perception Mode.

Okay. And then what happens next? A little bit of moderate noise.

A little bit of moderate noise. Pretty quickly, you will go into better ear mode. That is, we’d put a directional beamformer, so 4-mic beamformer on one ear.

Well, it’d be 2-mic beamformer then, right? Nope Just one ear? No.

All 4 ears still? It does use all 4, but it’s only on one ear.

Oh, that’s really interesting.

Yes.

OK, so you have very good control of the beamforming because you’re using both sides.

Yep.

But it’s only beamforming— the playback of the beamforming is in one ear only.

One ear only.

My better ear.

It’s actually on the ear where the side would be the most noise. OK. Yeah. If the noise is diffuse, it’ll go on your right ear. But it’ll go mostly to make sure that the noise on the side that has the most noise on it is reduced.

So that’s what you mean by better ears, not your better ear or your louder ear.

No, we’re going to make it your better ear.

Yeah, OK. So we can dynamically change which ear is going directional and which one’s staying omni.

Yep, absolutely. And that’s how a normal hearing person hears in noise. We use our better ear, the ear that has the best signal-to-noise ratio. That’s how we hear in noise, especially in cocktail party noise, you know, you’re just kind of leaning in and it’s the ear that gets the best signal-to-noise ratio, that’s the one that you use. And you need to hear all the noise around you in order to get the natural suppression of noise that our brains do. And so that’s what we’re doing. We’re trying to give you the better ear impact, just like a normal hearing user, and the ability to have everything you need to have natural noise suppression.

Okay. Now it gets crazy loud. Maybe not loud enough for the DNN, but it’s louder. What happens next?

Eventually, if the speech is all from the front, no matter what the level of the noise is in the background, if the speech is all from the front, we will then turn on both binaural beamformers and we will switch into the deep neural network. The speech has to be at the front though, otherwise we won’t turn it on.

Okay, so if I’m sitting at a table, 4 people say, even if it’s really loud, you won’t use the DNN because your DNN comes after the beamformer. So it’s always DNN engages forward. You don’t want that happening if there are people at your table to the sides.

Yeah, because we want you— if there are people to the table at the side, we want you to be able to engage with them. And the minute we turn on a very narrow beamformer, this is what you’re hearing is what right in front of you. If you want to engage with the people over here, you know, we want you to be able to do that. We want you to know that people are talking to you from the sides. But if you, you know, if you’re in If you’re in a conversation, if you and I are at dinner but there’s two other people, but you and I are talking, and so it will pick up that speech from the front, it will engage the deep neural network.

Is the beamformer operating exactly the same between Sensia and Vivia?

The beamformer is not acting the same exactly. What’s different? What’s different is with the deep neural network in Vivia, the beamformer is a broadband beamformer all the way to the lowest frequencies. The beamformer now in Sensia in the automatic program actually keeps the lowest frequencies omni and then has the beamforming above the real low frequencies. So then a very, very narrow beamformer in the mid frequencies and then above 5,000, the beamformer will open up a little bit so we can get more spatial hearing. We do that because you, in an automatic program, you want everything to be very transparent. And if we have a beamformer in there that goes to the low frequencies, it’s a very jarring change between the different, you know, the different omni programs to the directional programs. So we want the loudness to be very equalized and the change to not be dramatic. It will be dramatic in the fact that you’re going to hear that noise go down. But we don’t want that to be a jarring change. We just want it to be very natural and you just hear.

Okay. And if I’m engaged in a conversation with multiple people, then the DNN will not engage or will engage?

It depends on if you have a lot of voice right in front, it would engage.

It would engage.

Yep. It would engage in that case. So if you’re looking at people and it’ll turn on and it stays engaged for quite some time if it does turn on. Because again, we don’t want there to be jarring movements. But if we feel like the speech is in other locations, we want you to pick up that speech in those other locations. We never never want to put you in a situation where the hearing aid is making the decision for you. And unfortunately, turning on a DNN without a beamformer will put you in a situation that if there’s louder speech in the environment, no matter where it is, the DNN will pick up that prominent speech and take everything else out, which, you know, that’s a problem. You don’t always want to listen to the loudest person. You want to listen to the person you came with. And so we never want to put someone in a situation that that’s going to happen to them.

Yeah. And I mean, that was proven even in the Nexia where you had a good beamformer.

Yep.

But you mentioned music, that you’re not classifying music. So tell me about how it works with music.

Yeah. So if you classify music, the classifier gets confused. Because if you go into a restaurant and there’s music playing in the background, is that a music environment or is that a speech and noise environment? Most people will tell you that’s a speech-in-noise environment. I went there to listen and talk to the people I came with. The background music is background music, but most hearing aid classifiers will classify that as a music program or as a music environment. So it will not turn on directional microphones. It won’t engage noise reduction or deep neural networks. It does the opposite. You know, we will give more gain in the low frequencies and a broader frequency bandwidth and Everything you don’t want in a noisy environment, it will do. And that’s a shame. Those are the kinds of decisions a hearing aid should not be making. So we take music out of that equation and we never classify music. If you want to listen to music, that’s an intentional thing. You go to a concert, you go to the symphony, you are intentionally there to listen to music. We think you need a music program, you know, a separate program for you to listen to music. You can tune it up yourself if your audiologist lets you. You can do whatever you want with it. Right.

And you don’t do nonlinear processing and this sort of thing. The music quality is the best.

Sure.

You can— I think that makes perfect sense, right? Because I could go to, say, a jazz club, and I want— I’m with my wife, but we’re there primarily to listen to the music. I go music mode. But I could be in a similar situation, but it’s background music, and I want to talk to her. So since it’s impossible to identify what the person wants, what you’re saying is, if you want to listen to the music, go to music mode.

Yep.

Otherwise, we’re going to assume you’re in a speech and noise and concentrate on the vocal intelligibility.

100%.

Okay.

100%. And I think, you know, that’s a conversation that I really hope will come out of the launch of Sensia, because we really want to bring to the forefront that classification matters. And if you are going to have a hearing aid that you’ve spent all this money on and you have all these great features, they need to turn on. When you need them. And they’re not always turning on when people need them. So I think, you know, with Sensia, you know you’re going to get all the features you need in the environment you need them, because we take music out of the equation and we go ahead and give you speech in noise, the best speech understanding. That’s why people wear hearing aids.

And you’ve got some evidence that shows what percentage of the time the Sensia engages the modes you wish it did, right?

Yeah.

Tell me about that evidence.

Yeah, so we’ve had a couple We had one clear back in 2017, which was a very simple study. We put the hearing aids in a test box and we just played different background. So it was speech and quiet, speech and noise, traffic noise, and then we would just read out the data logging. And so from the data logging, you know, we knew what it was exposed to. Then we read out the data logging and compared it. And we were very surprised at those results. We were like, wow, a lot of these hearing aids are not classifying things correctly. And then there was another study, a peer-reviewed study, Yellum study, that basically said music was the problem. The more complex the situation got, music in the background, these classifiers weren’t classifying it correctly. This was a peer-reviewed published study. So did not come out of a manufacturer. Then we actually looked at our classifier a little bit more and looked not only at the classification, but do we turn on the right features at the right time? And we’ve recently— two of my colleagues have done a study on that showing that if you just classify very simple environments, speech, noise, or speech and noise, that most classifiers aren’t getting them right if there’s music in the background.

OK.

Without the music, we’re 100% We don’t classify music, so we’re classifying 100% correctly in those 3 simple environments. But everybody else classifies music, and they’re getting them wrong at least half the time.

Okay. So it’s interesting, right? Because you get better performance by taking a feature away, if you will, by abandoning music because it’s impossible to know a person’s intentions toward that music. So you just abandon music, and therefore you’re able to classify everything else more accurately. Yeah. Okay.

Yeah.

Okay. And also part of this announcement is a deeper collaboration with Cochlear. Tell me about that.

Yeah. So, you know, we have had the, the Smart Hearing Alliance now for, for a while. What we’re getting now with Cochlear would be the ability to use one remote control and everything will stream any of our accessories and such. Everything will go at the same time. You used to have to program one remote control for one and another. Now everything will be seamless and it will come together. These are hearing aids specifically made for bimodal. So you’ll be ordering a bimodal hearing aid now and you’ll be able to use just one remote control.

So when I get a CI from Cochlear, they would set me up with the bimodal version of Sensia.

Yeah. If that was what the audiologist chose.

Right, if that’s what was called for, right?

Hopefully, yep. You would get the bimodal Sensia. And people have been waiting for this for quite some time so that these things can be married together. You don’t have two remote controls. You can make everything work together.

So I can use one remote control, control both devices. I can use one multi-mic. It will play in both ears. For cochlear users, this is good news.

Yeah, it is.

So what else should I know about the Sensia? What else should people know?

Yeah, you know, I think the most important things to know about Sensia are accurate environmental classification, accurate feature engagement. We’ve integrated that deep neural network where you need it, when you need it, at the right times. And we still have the battery life we had before in the smallest AI hearing aid.

So what’s the battery life?

It’s at least 24 hours.

So more, more than a full day.

Absolutely. And engaging that DNN when it needs to be engaged. And we do engage that DNN when it needs to be and still have 24 hours of battery life in a very small cosmetic package.

And still waterproof.

It’s IP68. I wouldn’t say waterproof, but darn close.

IP68, good enough. I go running in the rain and I get hosed by a thunderstorm.

Yeah, you won’t have any problems. Nope, you won’t have any problems.

OK, terrific. And size-wise compared to Vivia?

It’s the same.

But then you also have the larger one still with a telecoil in it as well, right?

Yep.

So you can have telecoil and Auracast both.

Absolutely. And you can always have telecoil with us because our MultiMic has a telecoil in it. So you don’t have to buy the telecoil hearing aid. You can just buy the MultiMic.

Right. So if you’re comfortable carrying the accessory around with you, you can do Auracast or do loop that way.

Yep.

But if you’re not, then you can still get it in a hearing aid.

For sure.

All right. Well, thanks a lot. I appreciate you spending some time with me.

Yeah, thank you.

Always great to see you.

Yep. Thank you.

I have here with me Brian Dam Pedersen. He’s the CTO of GN Group. Thank you very much for spending some time with me. It’s the first time you and I’ve had a chance to talk.

Yep, thanks.

So you’ve had a wide and varied career here at GN. Tell me a little bit about all you’ve done.

Yeah, I think, I mean, that starts many years ago. I started GN already back before 2000 doing software and did that for many years until we embarked on our 2.4 GHz wireless system development. I worked on that for 3, 4 years and then transitioned into leading the team that implemented Apple’s MFi protocol here, and after that, the Google Azure protocol. So I’ve been very much involved in our wireless development here for the past many years. And for the last 8 years, I’ve served as Chief Technology Officer leading our research and system architecture groups here.

Okay, so quite a background both in the communications and then also in the architecture.

Yes, definitely.

Regarding the architecture, in a recent podcast I had said that we had mostly run out the audiological methods of improving speech in noise performance. In other words, you do all the beamforming and filters and whatnot. That turns out to be not quite true because even between the Vivia and now, you’ve made some improvements from that point of view.

Yes. So you could say when we look at the classic signal processing, it is true that we sort of have been on a curve where over the past 20 years we have seen rapid improvements, especially for speech in noise, both with beamforming and signal microphone noise reduction. And that is sort of leveling out. It’s not that we can’t find any more improvements, but it is getting increasingly difficult and it’s increasingly more expensive to find that extra dB. That was easy 10, 15 years ago. It’s harder now. But you could say with the advent of deep neural networks and machine learning and getting the ability to actually put that into the hearing aids, we simply open a new toolbox that will allow us, I think, to take another round of rapid improvements that we’ll see over the next years before we then also will see that level out in 5, 10 years. So I’m really I’m really positive about what we see within, within AI and what that can give together with the classical approaches.

Agreed. And that’s, that’s what I meant was the classical approaches, signal processing, classical signal processing that I had thought would pretty much been tapped out. Yeah. Then, you know, increasingly powerful DNN chips and, you know, edge AI models to go with it are the next level.

Yes.

But you’ve managed to get some improvement with classical techniques too.

Yeah, that’s true. I mean, it’s not like this is over, right? I mean, we are still doing improvements to our beamforming technology, and we are still doing improvements to the underlying algorithms there, and also the hardware. The improvements that we have done over the past years is a mix of, you would say, classic algorithm improvements and software optimizations, tuning, but also tuning our hardware very specifically to be able to support the things that are important for beamforming. Highly aligned audio between the ears, precise low noise transfer of audio between the ears. Those elements that you need to really do this very sharp, very narrow beamformers that enable us to do better speech in noise today.

Well, and I’ve seen devices that have sharp beamformers that tend to create artifacts.

Yeah.

Especially on the edge of the beam. Yes. You’ll hear distracting artifacts that can wear you out in time.

Yes.

But you have apparently solved that problem, from what I understand. And how then are you able to operate a variable, sometimes quite sharp beamformer while still making a comfortable listening experience?

Yes, so that is this multiband approach that we have to our beamformer where we don’t try to do everything across the entire frequency band. So we have a We process the low frequencies separately, basically in an omni mode, because there’s not much gain in the doing beamforming there, we can’t do that due to the large wavelengths. So that is basically attenuated with classical single channel noise reduction. Then we have a speech focus area where we do everything we can in order to enhance the signal-to-noise ratio with the 4-microphone system. And then we blend in a broader, less aggressive approach where we go more monaural and stereo in our beamforming technology in the higher frequencies. To preserve spatial cues up there. And when you mix all these together, you also erase some of these very sharp edges that would be around this very sharp beam, simply because there will be a mix-in from one of the other bands when you do these crossovers. So that tends to soften off the sound picture and make it a lot more natural than we otherwise can do. So if we did a full-band beamforming, beamformer, you would get a mono signal, basically. You would be mono listening to the person in front of you. But it turns out when we remove all these spatial cues, we lose around 3 dB in SRT, and that is more or less what we gain on the beamformer. So basically—

You give everything away, right?

You give everything away that you just gained, but by doing this mix-in of the spatial cues, You can actually use that extra SNR also because the brain is getting a more natural signal that it’s more used to listening to. So therefore, the benefit we see is a lot higher when we do that.

Okay, so you’re always operating a beamformer variably by frequency, low, middle, and high.

Yes, yes.

Which then strikes me as if I’m, for example, talking to you in a restaurant and it’s in full beamforming mode, server comes up to the side, I’m still going to hear them, right?

Yes, exactly.

I mean, I’ll have to turn my head.

Yeah, but you will notice there, you’ll hear their speech there, you will also be able to hear it is on the right side of you. So it’s not like you will go, okay, where’s that sound coming from? You will know that that’s coming from over here.

Okay. So it strikes me as being a pretty natural experience while still increasing the SNR. We haven’t even talked about the DNN yet.

Nope.

And so let’s do that because I guess the question I have is, is there any difference in the DNN between Vivia and now, or is it essentially the same? What has been done there?

So on the DNN In the end, it’s the same architecture we are running. Of course, we have taken some learnings in how we tune the system and make it collaborate with the beamformer between Vivia and Sensia. So it’s not exactly the same system, but it’s largely the same system, right? We have done some changes to the way that, especially for the automatic mode, to the way that the DNN engages, because when we kick in a technology like the DNN where the sound picture gets so different, you don’t want this experience of people saying, “Okay, something changed here,” right? I mean, that’s a fairly disturbing thing. So, we don’t want a very sharp transition between running in the normal speech and noise mode and then kicking in the DNN. And that has taken some changes in how we actually integrate the DNN with the rest of the system. But the core DNN, is by and large the same as we had in Vivia.

Right. Then it’s really model development that you have done and coordinating with the new scene classifier.

Yes, exactly.

So yeah, and as far as the scene classifier goes, I mean, I always thought the Vivia scene classifier was pretty good.

Yeah.

What have you done under the hood to make the scene classifier even better?

Yeah, that’s again a matter of tuning, and then we have introduced the capability to better rely on spatial information, where speech coming from. Again, the idea with the DNN in the beamformer is that when you engage this beam, you still have a cone of noise in front of you, right? That cone of noise is what the DNN is being asked to clean up afterwards. And that only makes sense if we can actually see that, yes, there is speech in front of you and we have some noise that needs cleaning up. Otherwise, there’s no need to engage the network. But the state technology is in right now. it is still fairly draining on the battery to engage the DNNs. So we don’t want to do this willy-nilly. We don’t only want to do it when we actually can see there’s real benefit. So extending the classifier to be able to also find these scenes where it’s actually beneficial. So when we think about classification, we think less in terms of absolute sound environments and more in terms of sound cues that can steer our algorithms. So we are looking for speech in a certain direction in order to set up the beamformer system to optimize for that, right? We are looking for noise level in order to steer the depth of the noise reduction to make sure we are not too aggressive so that people get disconnected from the environment, but also provide enough so that you actually get a comfortable noise level in all situations. So that’s how we think about this.

So that implies then that one improvement that was made was that you’re better able to recognize speech versus everything else.

Yes.

Because then you use that to adjust the beamformer. Yeah. Because you can go wide or narrow according to where voices are in my environment.

Yeah, yeah, yeah.

Which is something new then, correct?

Yes, exactly. Okay. Exactly.

And so is there anything else that people should know?

I think the one feedback we got from Vivia was exactly, okay, but there is this great speech-in-noise mode with the DNN. But we have to train people to go into a special program in order to use that. So just getting this automatic mode means that we can make this more available to more people, right? And I think that in itself is a big step forward. We are approaching a time where, except if you are in very specific environments like listening to classical music or something like that, you will be able to stay in your Program 1 and the device will largely do what you would expect it to do in order to optimise the sound picture. And ultimately that’s the goal. We don’t necessarily want people to go around and fiddle with their devices all the time. If we can get the device to do what you expect it to do in all the scenes where we can reasonably let the device argue about this, that would be the optimal situation. So, in that sense, from a set-and-forget point of view, this is a step up from Vivier.

And I think that’s an important improvement. I’ve gotten spoiled by the scene classifier, right? It works well. But you’re right, you have to pull out your phone if you want to go into full DNN mode with the Vivia And that’s fine for me. I don’t mind. It’s like, gone. But I know people who don’t have smartphones. And so they either have to remember how to cycle through the program or they just end up not using it. And so the more automatic you can make it— I mean, it’s convenient for everybody, right? You just go through life and not change things.

And then it’s also nice to be able to bring this to more people. We’re introducing the manual DNN mode also now in price point 7. So now we can broaden it out to even more price points. And it is still— we have the extra chip in there still. So it is a more complex device to build than the normal hearing aids. But we have been able to do some optimizations around our electronics integration that allows us to actually put it into low price points as well.

Okay. I can say that with the Vivia, I never really noticed any increase in the latency when operating the Vivia. No. I’m assuming that it’s the same in this case.

Yes, exactly.

That the chip is fast enough.

Yes. The way that we integrate the DNN actually means that we don’t have any extra latency in the signal processing once we kick into that.

Okay. Now that you’ve done this, to the extent that you can share, what does the roadmap look like? This is now your second product with a DNN in it. You obviously have goals for the future. Like, where do you see further improvements coming from?

Yeah, as I started by saying, I mean, we see this as the next wave that can really improve, especially speech in noise. When you listen to a lot of the things coming out of academia and things that run on bigger computers than we currently can have in hearing aids, it’s clear that you can do incredible things with DNNs and speech enhancement. So to us, this is only the first step. I don’t think I can get into specifics on what will the next products be, but I think it’s fair to say that there will be significant improvements in speech-to-noise driven by DNNs in the coming years. And that would be across the industry. That’s not only for us. Everybody is putting massive research into that right now.

Yeah, and I think it’s wonderful. I mean, I’m a person who always has a hard time in noise. If I grab, say, a 5-year-old hearing aid and put in a modern one, it’s no contest at all.

Nope.

No contest at all. So I think that’s really exciting that you’re going down this path.

Yes. And I think that is what makes us go up and go to work every day, right, to make sure that the devices we deliver actually deliver meaningful benefits, even in these more challenging environments, and especially the The cocktail party, the restaurants, they have always been a challenge to hearing aids. And I think we have been able to take some meaningful strides into solving that over the past years and continue to focus very much on that.

Yeah, and that’s really meaningful because you’ve got the whole program to educate HCPs and give them materials on addressing lifestyle and the health benefits of addressing your hearing loss. Anything that you do to enable people to go out and enjoy social situations without being stressed, without being fatigued, where they can be comfortable and relaxed, is a terrific thing.

Exactly, exactly. And that’s what we would like to give back to people.

So yeah, from a personal point of view, thank you very much.

You’re welcome.

And thanks for spending some time with me.

You’re welcome.

As you heard, another exciting bit of news to come out of the announcement was the availability of bimodal versions of the Sensia and SuperPower Enzo IA to work seamlessly with Cochlear CIs. That will offer important benefits to bimodal CI and hearing aid users. I’ll report more on that as additional details become available. In the meantime, thanks for watching or listening to this edition of This Week in Hearing.