Action Audio Ball-Tracking Tech Lets Sight-Challenged Fans ‘See’ Tennis

Introduced at Australian Open, the system converts officiating data is into audio cues

Last month’s Australian Open attracted a lot of attention during its two weeks of tennis, chiefly for the Novak Djokovic imbroglio. But the first slam of the year was significant for another development: the debut of Action Audio, a new ball-tracking technology that enables blind and visually challenged fans to experience the game in a new and more intense manner, using sound to interpret ball and racquet cues.

Tennis courts on the pro circuit use line-and-ball–tracking infrastructure comprising 10-12 cameras to collect data to support official calls on whether a shot landed in or out of play. Action Audio uses that infrastructure, which can track the ball’s movement at nearly 50 times a second, and converts the data into 3D sound. The process takes less than a second, allowing it to be broadcast in real time alongside live announcer commentary.

Aural Cues Based on Familiar Sonic ‘Languages’

To aurally indicate how close the ball is to the sideline or baseline, the data is processed into a series of audio beeps derived from existing sonic “languages” familiar to blind and low-vision audiences. If the ball is close to the line, three blips are heard; if it lands farther inside the line, two blips; one blip means the ball was hit toward the middle of the court.

AKQA’s Tim Devine: “Action Audio lets the listener create a picture in the mind’s eye from the sounds.”

In addition, the sounds are processed in stereo, with the beeps’ left-right speaker placement reflecting the side of the court the play is on. Further, as the ball moves around the court during play, it produces jingles and rattles, loudest when the player strikes the ball and fading gradually as the ball travels. Tonality also is used: high-pitched clinks indicate a forehand; low-pitched clinks indicate a backhand.

Created by a partnership of Tennis Australia, Monash University, and UK-based digital-design and -communications agency AKQA, Action Audio was available as an online audio stream across every match at the Rod Laver Arena via Australian Open live radio and through Google Assistant. It was the first-ever major sports competition to provide such an experience for blind and low-vision fans. It had been beta-tested at the Australian Open 2021 tennis finals, following a two-year process of co-design and prototyping with members of the blind and low-vision community.

“The challenge was really about, how do we give people more information in a really simple form? And the whole point of Action Audio is to give [visually challenged] people more information that allows them to make their own appraisal of what’s happening on the court,” says AKQA Executive Innovation Director Tim Devine, the project lead on Action Audio, from his office in Queensland, NZ. “Obviously, there are radio commentators who talk in real time and as it’s happening, but they are reducing the information to their view of what’s happening. Action Audio lets the listener create a picture in the mind’s eye from the sounds.”

Action Audio technology debuted as an online audio stream at the 2022 Australian Open but can be synched with picture and audio for broadcast or streaming.

The Action Audio feed for the Australian Open was available as a synchronized stream, although it was used only on radio for this event. Devine says any broadcast or streaming service can access it and sync it with picture and audio.

“It’s effectively another audio stream that we can overlay on any broadcast,” he explains, adding that it is also accessible via Google Assistant, a platform often used by the blind.

Headphones are preferred to fully experience the Action Audio experience, which is designed to work in a binaural mode, mimicking the ability to locate sound’s directionality. Devine notes that Apple’s AirPods can accommodate the head-related transfer function (HTRF) response, which characterizes how the ear-brain connection receives and processes a sound from a point in space, and can interact with Action Audio’s binaural capability.

A Future in Sports Broadcasting

Devine says the company is currently actively seeking partnerships, including with sports broadcasters. He believes that the technology underpinning Action Audio can be extended to a number of sports. Many, he points out, are increasing the amount of data they can generate as part of their normal operations, such as the Hawk-Eye line-calling system used by the NFL and USTA and the VAR play-review system used by Premier League and other soccer leagues.

“Data is going to continue to grow around all sports and become more accessible around more sports, and the audio technology is going to grow around it,” he says, suggesting that Action Audio is one way to use that data as a portal to an augmented-reality sports metaverse.

“The immersive aspect of these spaces requires a significant investment in audio streams,” he adds. “Action Audio is part of that. It’s designed for blind sports fans, but it can open the door to much more.”