Olympic Reflections: NBC Olympics’ IT Effort Drives New Production Workflows
When NBC Olympics began its efforts to keep more of the production team out of Tokyo, NBC Olympics VP, IT, Dan Robertson and his team helped the organization shift into overdrive to create policies ranging from how the networks are shared and managed to who has access to what type of network. The good news, he says, is that the COVID pandemic already had everyone used to working in new ways.
“COVID prepared everybody in the company to collaborate in a structured way,” says Robertson. “We use collaboration platforms heavily across the enterprise and standard client technologies. The pandemic had everyone working remotely and collaborating outside of having to be in a conference room, for both admin-type work and production. It prepared everyone so that, when they got [to Tokyo], it wasn’t a new practice. It was natural for them to come in and just be able to continue collaborating that way.”
As broadcast services and products became IP-based, Robertson notes, the need for and use of IT circuits grew, not only in Tokyo but also in Stamford, New York, London, Miami, and other locations.
“The scrutiny and the dependence on networks and some of the traditional IP stacks certainly weighs a lot more on how we deliver,” he says. “The broadcast engineers are learning a lot about IP stacks and what it takes to make things not only function in a routed type of environment but emulate deterministic workflows.”
Historically, the relationship between IT and broadcast engineering can be tense, but, with nearly every workflow having some interaction with networking, a conversation is created around network connectivity, bandwidth determinants, and security.
“I’ve seen the relationships between IT and broadcast in prior jobs,” says Robertson, “and, at this Olympics, it has been an incredibly healthy and coordinated effort. The team here tends to function more like a family that’s away from the mothership and [has] to get things done and lean on counterparts to make that happen. So it has been a very healthy and constructive relationship as the engineering and IT teams have commingled to bring solutions together.”
For example, Mike Muffoletto, manager, NBC Olympics IT and On-Air Systems, NBC Sports, notes that the teleprompting workflow changed dramatically, with most of the operators based in Stamford and the talent located in Tokyo at the IBC and the studio at the Odaiba Hilton. “We based our hardware within the control room at the IBC, and the engineers essentially routed the teleprompter wherever it needed to go. It was a whole new workflow to expedite and streamline the operation.”
It also allowed the team to reduce the number of people onsite at the various studio locations, an important consideration given the COVID pandemic.
“We become not only the technical experts but the operators,” he says, “using MiFi wireless routers to connect to iPads mounted on cameras and laptops that provide the scripts. It has been a great experience streamlining a lot of the workflows.”
Another way the pandemic has changed production is that people have grown accustomed to having more and more devices at their fingertips when they are at home and working remotely. When they came back on location for the Tokyo Games, those devices came with them.
“Every person is walking around with four devices that they’re connecting at all times. They bring them here and want to use them here,” says Chris Lubbers, director, remote operations, NBC Olympics. “We had a lot of applications that we brought in for collaboration that we’ve never used, and it helped with the social distancing and the commentary positions. And I don’t see it slowing down.”
Security was paramount to connecting so many broadcast facilities, venues, and rental facilities at each venue (and beyond). According to Robertson, operating procedures for connecting rental gear or production trucks to the NBC network were established to ensure that connected facilities at venues were reconfigured and scrubbed with security tools to make them virus-free and reduce security vulnerabilities.
“We have enclaves of connectivity: broadcast environments, administration environments,” he explains. “There are various things that, depending on where they get plugged in, have to be scrubbed and certified before [being] placed into production. It is one of the more painful things we went through at these Games, as we needed to ensure that we could get connectivity off the network for the certification and the registration of those devices.”
To Lubbers, the biggest change in the last few years with security is that engineers at the remote sites are aware of the security challenges. “In the past, it was IT telling them, and they would say, who are you and you aren’t touching our equipment. But now, as we coordinate before the bigger events, they’re all on board. I think that’s the biggest thing: it has been more of a team effort.”
At the venues, Robertson says, NBC Olympics brought in its own networks, both wired and wireless. Devices needed to authenticate to get on those networks, although there was also raw internet connectivity for personal devices. “The raw internet goes through a DMZ at the IBC, but that is the place where people can jump on with their own personal devices.”
Mike Della Sala, manager, unified communications, NBC Olympics, notes that the server they would access in the DMZ in the IBC allowed the IT team to open or close access. “We didn’t see any issues with it, so it was pretty secure.”
Also helping in security was desktop as a service, with virtual desktops giving users full access to the environment and applications, provided they used one of the NBC Olympics virtual machines.
Wired connections also played a big role, especially in areas critical to the broadcast, talent interactions, or athlete interviews.
The 2022 Beijing Olympics will offer its own set of IT challenges — most notably, the inability to access certain tools that have become part of nearly every rightsholder’s toolkit. Broadcasters rely on things like cloud services and social platforms for managing operations, and, in Beijing, some of those services will not be available without dedicated private lines. The good news is that the Beijing Organizing Committee has said that anything done within the Olympic fence will not be affected.
Robertson notes, though, that NBC Olympics has a lot of operations that are outside the fence. “What will happen there is still to be determined.”