A few years ago I had the good fortune to work on an amazing project at work - Microsoft Soundscape.  My team at Microsoft joined the effort and delivered a truly awesome experience over the 8 months of work we put in.

You can see some of my work on the software side of this project here.  This post is about the other half of the work we did.

The hardware to make this experience work consisted of 3 components -

• iPhone
• Stereo headphones with integrated IMU
• A remote control

This project was created in Spring 2016, predating IMU integration into any commercial headphones and assistant integration into headphones.  Additionally, wake-word capability wasn't available on custom devices from any vendor (at least not without NDAs).

The project started as a Windows Phone application, but one of my first contributions to the project was to move development to iPhone and iOS (visually impaired smart phone platform preference 2014 86% iOS
2017 80% iOS.

So all that was left to do was to integrate an IMU into a pair of headphones and make a basic remote control that could be paired via bluetooth to the iPhone.  Easy right?

Designing and building an AR audio headset

The first headset - Aftershokz + Sensor Tag

After a significant amount of ethnographic research with Guide Dogs members, we had a very clear set of minimum viable criteria to a headset.

• It must appear like a standard commercial set of headphones.  Users did not want to wear or carry any device that appeared as a medical or special assistance device.
• It must not obscure environmental sound.  Blind users literally depend on their heightened hearing to get envrionmental cues for wayfinding and danger avoidance.

In addition to user requirements, we had several more:

• Must pair as a standard bluetooth device to iOS
• Device management (pairing, charging, storage) can be handled by the end-user, no IT specialists or care assistants.

When I joined the project, the team had been using the Aftershokz Sports M3 and attaching a TI CC2560 SensorTag to the back.

This worked in meeting the above requirements initially.  However a few problems in the user experience popped up that made us switch directions.

First was the SensorTag.  It has incredible battery life, close to a year on a button cell battery (and TI advertises this heavily).  What they neglect to mention is that this battery life claim is with the IMU turned off.  When you enable the IMU with a reasonable sampling frequency necessary for accurate positional audio (10hz minimum), the battery life plummets to a few hours.

This meant we had to provide instructions to blind users on opening the enclosure, removing and replacing a button cell battery up to twice a day.  Total non-starter for a real world experience.

Second was the Aftershokz. On the positive side, they delivered really exceptional sound quality for the time, and didn't obscure envrionmental sounds.  However, it turns out that bone conduction doesn't work too well with positional audio - not that it doesn't work, but accuracy and precision suffer in comparison to traditional headsets (over or in ear).  And more importantly to our users, BCTs (Bone Conduction Transducers) have a LOT of sound leakage.  Whatever you hear, so does everyone within arm's length.  This loss of privacy would also make others aware of a user's special use of the device, breaking their desire to blend in.  Last, the headset was wired and has a breakout box with a battery and switch, that a user would need to charge regularly.

I actually created instructions for this variant of the headset and remote, both visual and audio -

So, we needed something a little better.

Take 2 - Custom Bluetooth headset

For a lot of reasons, I can't go into the specifics of the board we built.  But we can talk about the industrial design.  Here are the final visual instructions for this solution.

First, we started with a commodity shell (most commonly referred to as the BS19C).  We only used the casing, but this was a crucially important decision - we didn't need tooling to prototype solutions.  These headsets are available worldwide, from hundreds of vendors, for almost no cost.

This gave us enough room to get all the electronics in, have a single rechargeable battery, a familiar form factor to wear and not appear as specialized equipment.

We were fortunate to have an awesome industrial designer - Oscar Salandin - to iterate on the earpiece design, landing on one that worked well in testing and was easy to position with good accuracy.  He did an amazing job producing dozens of options, that my colleague Michael Platt tested for efficacy and audio quality.

While I toiled away manufacturing the ear pieces, my colleagues worked around the clock a few miles away soldering PCBs, mounting and assembling units.

For more on the process of manufacturing these tiny ear pieces, see the article here.

We got the headsets working well, and they shipped on time for field trials to begin.  The last piece of the hardware puzzle was the packaging, but I'll have to cover that in another article in the future.