Live From Rio 2016: NBC Olympics Tech Leaders Reflect on the Massive, Successful Production Effort

‘I could not have been more proud of all that we as a team accomplished.’ — Dave Mazza, NBC Sports and NBC Olympics

The 2016 Rio Olympics have come to a close, and the NBC Olympics machine, for the most part, is being dismantled (a much smaller technical presence will remain onsite for the 2016 Paralympics). The end of the Games also begins the evaluation of what worked, how operations can be improved, and how the entire Olympics production and technical philosophy might change for the 2018 Winter Olympics in PyeongChang, Korea. But the end of the Olympics also provides an opportunity to step back and reflect on a technical team that enabled the production team to serve fans an unprecedented amount of content on an unprecedented number of platforms and devices.

NBC Olympics’ Dave Mazza says the production team was able to get to full speed quicker than usual for an Olympics.

“I could not have been more proud of all that we as a team accomplished here in Rio and [at NBC Sports facilities] in Stamford, Orlando, and Hialeah,” says Dave Mazza, SVP/CTO, NBC Sports and NBC Olympics. “In the run-up to the Games, there were obviously tremendous concerns about all sorts of things. But everybody kept their heads down, pulled together, and got their portion of the job done. We may not have been working on our Plan A, Plan B, or even our Plan C, but it is a testament to so many seasoned, flexible, and inventive professionals that we able to go up, over, and around most obstacles standing in the way of our success. People asked if we were going to water down our plans in light of the difficulties on the ground. We not only did not do that, but the entire team executed NBC’s most ambitious Olympics coverage plan to date, on all fronts: broadcast, cable, digital, and social.”

And then there are the freelancers, the ones who volunteer to be part of a project that, every time it occurs, basically produces an event in a city that has no experience with it. For all the veterans involved, it is, in many respects, a first-time effort.

“We couldn’t do it without so many freelancers who have worked with us before,” says Mazza. “And I guess they like it enough; they come back with a smile on their face and knowing that there are certain things, like transportation, that don’t work well at an event this size. They also know about the crazy way we have evolved over the years, so they can figure out new things without hardly even having to be told. And, this year, we have about 15 really good young guys in the mix.”

Adds Terry Adams, VP, IBC engineering, NBC Olympics, “The veterans here are A-level, and even some of the younger ones who are working their first Games had a skill set well beyond what we thought they would have. They were terrific; as far as our staff in the IBC, there were no duds.”

The NBC Olympics engineering team stepped up at the 2016 Rio games to meet the challenge.

The high level of performance also extended to the venue-operations team. According to Chip Adams, VP, venue operations, NBC Olympics, the veterans involved in venue operations once again delivered.

“We also found some great new tech managers to roll into places,” he adds. “It was difficult to get some of our regular tech managers in because of the payload that NBC Sports now has [with] the Open Championship, Sunday Night Football starting soon, etc.”

One of the consistent themes for every broadcaster at the IBC was logistics, and NBC Olympics was no different. Throughout the Games, everyone’s operations were hampered by issues related to power, official bus transportation, traffic when not using official transportation, language barriers, and more.

“With the trucks coming in sort of late and with some of our shipments being delayed by customs, we were pressed to get into the venues with the amount of time we would typically want, which would be a couple of days,” says Chip Adams. “And, this time, with all the construction going on and the [port] where the trucks came in, it took them a day to get to the venues. So it was just a challenge getting everyone up to speed and moving everything in on a truly fast-track install schedule.”

He adds that the NBC Olympics venue-operations teams, across the board, stepped up amid some challenging conditions. “We have good teams all around, and they are attentive to what production wants to do. Their main goal is making sure that the production teams get what they need to have the best production on-screen.”

One production group affected by the way the Rio 2016 venues were laid out was the diving team. With the diving and swimming events in separate venues, two separate setups were required, and the diving production team was unable to leverage some of the equipment deployed for swimming. For London 2012, for example, the team had access to two RF cameras, six edit suites shared between diving and swimming, and five EVS servers. For Rio 2016, the EVS complement was reduced to two, and the editing was done at the IBC.

The fact that the Opening Ceremony and the athletics were not in the same stadium was another issue, because traveling between them during setup basically was a full-day trip.

The distance between the IBC and the studio at Copacabana Beach also caused some logistical headaches.

“There are certain support services you need to maintain a studio,” Chip Adams notes. “When the studio is here at the IBC, there is space, access to writers, and a lot of support services that are a challenge to get on a beach that is being eaten by the ocean.”

Overall, he adds, the venues themselves were in pretty good shape when the NBC Olympics arrived in force.

“What [the venues] weren’t ready for,” Adams adds, “was some of the overlays we had: tech, power, and data. So those three things were late, and that was endemic across the entire 2016 Rio project. But, facility-wise, we never had any complaints about being unable to get camera platforms in or having them built. All of those sorts of things were ready for us. And the Rio venue managers and the organizing team and the OBS venue team were very accommodating in helping us get what we needed done.”

Time for Reflection
For Rio 2016, the NBC Olympics team tackled a number of new workflows and also delivered an unprecedented amount of content, both on TV and online. The team also, arguably, does more to add its own production touch and flair to its content than any other rightsholder does.

“We will look at the degree of difficulty and the complexity of things, and that usually means figuring out how much risk something put us in,” Chip Adams explains. “Of course, just because something is hard doesn’t mean we won’t do it, because that’s what we are paid to do. But sometimes it gets so complex, we are on the edge of a cliff.”

Much of the risk and complexity in a project the size of the Olympics is because more and more of the technical and production infrastructure cannot be evaluated until it is in actual show mode.

“The time to configure the systems now exceeds the time it takes to wire it,” Mazza explains. “That is very different from the past, when it would take 60-90 days to wire everything.”

Terry Adams elaborates: “The physical stuff is not the hard stuff anymore. It’s watch folders, data paths, and naming conventions for all of this huge amount of stuff that is connected.”

That is one of the reasons it was so important for NBC Olympics to preconfigure the Rio 2016 system at NBC Sports headquarters in Stamford, CT. Preconfiguration took place on the floor below the Highlights Factory.

“In Stamford.” Mazza points out, “we have the luxury of a maintenance staff, and we can have a meeting in the morning and then go downstairs and work on configuring the system. For previous Games, it was a 90-minute trip to the Olympics field shop, so a full-day trip is replaced with a quick trip downstairs.”

Terry Adams explains, “The whole thing is now file-based, so you have to have this great data structure in place. We were able to do that before we left for Rio, and essentially all we did was make that fiber connection between upstairs and downstairs 4,000 miles longer.”

The continuing challenge for NBC Olympics and every other rightsholder at the Olympics and even for OBS itself is the transition to IP-based workflows. A 17-day event is entirely different from any other sports production, and a move to an all-IP infrastructure is still a few years off and requires a new way of thinking and problem-solving.

“Troubleshooting in IP is an entirely different skill set, and it’s going to take a while for the broadcast guys to move into an IP world,” says Terry Adams. “An IT switch taking 15 minutes to boot just is not acceptable.”

So there is plenty to consider with respect to the 2018 Winter Games, set to be held in PyeongChang. The nature of the Olympics schedule means that there is a little more than 18 months between now and those Games. The Winter Games are also a bit smaller. And then, of course, there is the massive time difference, with PyeongChang 11 hours ahead of NBC’s Stamford facility.

“While South Korea does have a reputation for modern facilities in its cities, that is not where we’re going,” Chip Adams points out. “We will be in the middle of the mountains. So we will have to do a lot of due diligence with respect to power and data connectivity. Plus, we will need to be more buttoned-up and do more of our integration prior to getting there, as there are not a lot of Radio Shacks or cable vendors in the middle of the mountains.”

Adds Mazza, “It may not be visible at home how many of the things at the Olympics work, because it all is intended to look like any other high-end truck-based sports-event coverage. A huge credit for that goes to OBS, who does a tremendous job of absolutely first-rate host coverage.

“For NBC,” he continues, “the complexity that goes into customizing the coverage of all 35 sports venues comes in two flavors. First is the logistics and local conditions of the larger venues for the four remote-production units, four  flypacks, and the huge IBC setup that we had here. Second is the complexity of smaller venues, using the mixed zones, commentary positions, remote-controlled switchers and splits, 23 announce booths, 13 control rooms split across four cities, and all the circuit routing in both directions to allow the production to work seamlessly from two continents. Even for those of us who have grown into this over the years, it is still somewhat mind-numbing in complexity. Add to that, that it all was working in real time, live. …. Suffice it to say … we are very relieved — and somewhat tired.”