In Age of Coronavirus, Sinclair, Altitude, and AT&T RSNs Test MTVG ‘Cloud Control’ To Customize Feeds Remotely

New remote-production platform is seen as ‘next evolution in the dual-feed model’

Although no one in the industry can be totally sure what live sports productions will look like when major U.S. leagues return to play, one thing is certain: the footprint and workflow of the production compound will have changed. Instead of the typical setup for an MLB, NBA, or NHL compound often handling three concurrent productions (two RSNs and a national broadcaster), many leagues are pushing for a world-feed model that would reduce the number of people onsite in an effort to maintain social distancing. However, this model would limit RSNs’ ability to customize broadcasts for their team partners and passionate fanbases.

With that in mind, Mobile TV Group, which led the innovation of the dual-feed model that has allowed RSNs to create customized telecasts for years, has launched Cloud Control.

Altitude’s producer/director at its NOC was able to call the game as if he was in the truck during MTVG’s Cloud Control test last week.

The new live workflow enables production personnel normally located in the mobile unit to work at their network offices and remotely control equipment and resources housed in the truck. Cloud Control can be used to produce multiple feeds of a game (world feed, home show, visitor show) with just three or four people per feed in the mobile units. MTVG’s dual-feed mobile units (A and B unit) can produce up to two feeds with Cloud Control with the potential of a third feed.

“We believe this is the next evolution in the dual-feed model and will allow us to produce a game broadcast focused on the hometown team that will satisfy the needs of our [team partners],” says Mike Connelly, EVP, Sinclair/Fox Sports Regional Sports Networks. “It keeps our staff close with our teams in our home markets instead of a centralized control room and also allows us to maintain the high quality of a dual feed, which has been one of the biggest game-changers in our business.”

Last week, MTVG successfully conducted tests of Cloud Control in Denver with Sinclair Broadcast Group/Fox Sports North, Altitude Sports, and AT&T SportsNet Rocky Mountain. The production had immediate control and use of all facilities in the mobile unit, including cameras, EVS replay, super-slo-mo, graphics, scorebox, intercom, and audio with less than 100-ms latency.

“The producer and director were able to call the show as if they were at the venue even though they were at their offices,” explains Phil Garvin, founder/CEO, MTVG. “They said it felt just like being in the mobile unit. Essentially, we’re taking all the [crew] that is usually located in the [visitor mobile unit] in a dual-feed model and moving all of that back [to the network’s offices]. If you want it to be exactly the same personnel, same level of cameras, replay, and super-slo-mo that you have today, you can do that; the crew is just in a different location.”

RSNs See Cloud Control as Viable Alternative to World Feed
As MLB, NBA, and NHL explore options for resuming play in the coming months, RSNs are looking to maintain some version of the dual-feed model in which home and visiting production teams are able to customize telecasts for their respective markets.

MTVG sends back 1080p multiviewer feeds to enhance the resolution for the producer/director back home.

“One of the biggest issues in the [RSN] business is the intimate relationships between us and our team partners,” says Connelly. “We’re embedded with these teams to tell their side of the story. Simply put, the world-feed model doesn’t allow us to do that. If we get into a REMI model, centralized control room, or world-feed situation, that significantly limits our ability to tell that team’s story. This [Cloud Control model] solves that problem.”

Ken Miller, GM/executive producer, Altitude Sports, adds, “If the choices are world feed or this model, then it’s not even close: we choose [Cloud Control]. We have spent a long time perfecting the dual-feed model, and it would be difficult to give that up. A world-feed broadcast typically plays it down the middle, and that isn’t going to cut it for what we need.

Notes Erica Ferrero, senior director, AT&T Sportsnet Rocky Mountain, “Accessing truck assets from our home studio with minimal latency allows us multiple ways to approach a broadcast.”

How It Works: Inside Cloud Control
Cloud Control uses multiviewers, intercom, and control data, rather than sending camera feeds to a central control room as previous “at-home” systems have done. MTVG sends a 1080p multiviewer feed to enhance the resolution for the producer/director back home.

The production had immediate control and use of all facilities in the mobile unit with less than 100-ms latency.

The only crew members onsite are an engineer, TD, A1, and V1. The producer, director, graphics/scorebug and replay operators, (possibly) talent, and all other personnel are located at the broadcaster’s offices.

The system allows almost unlimited camera and replay resources without the need for a high-end control room at the broadcast center. Cloud Control can use a standard office space with a 250- to 500-Mbps circuit to the mobile unit at the venue. MTVG provides networks a list of required equipment (a one-time cost of approximately $20,000 plus $10,000 for each EVS operator position) or will install it on a lease/rental basis (mobile units must be Cloud Control-compatible).

The Test: Cloud Control Goes Live
After more than two years of development, MTVG conducted a series of Cloud Control tests the week of May 11 with Sinclair, Altitude, and AT&T SportsNet. MTVG’s latest two-trailer, dual-feed mobile unit — 47 Flex and 47 VMU — was onsite at the Pepsi Center in Denver, and, although no actual game was taking place, several cameras and camera operators were deployed.

In each office, producer and director were seated in front of large monitors featuring multiviewers with up to 18 images (fully configurable by the user)

The mobile unit housed only four people: the TD, an A1, a video shader, and an engineer. Producers and directors were located at the offices of Fox/Sinclair Sports North (Minneapolis), Altitude (Colorado), and AT&T SportsNet Rocky Mountain (Colorado). Since the Pepsi Center was less than 20 miles from Altitude’s Network Operations Center, the RSN’s transmission provider, The Switch, routed an additional 1,000 miles into the feed to better replicate an actual away distance. Despite the transmission path, Altitude engineers found only a 15-frame/90-ms delay.

In each office, producer and director were seated in front of four large 55+-in. monitors, each with multiviewers with up to 18 images (whose selection, size, and positioning they could determine). Each position had full intercom panels (RTS KP-5032), giving the producers and directors exactly the same communication capability they would have in the mobile unit despite being up to 3,000 miles from their TD and A1.

“We’ve been testing different versions of this [model] with MTVG for a couple of years,” says Fox Sports Networks VP Steve Grigely. “The biggest goal has been to make sure that the production team is comfortable, because, if the front bench isn’t comfortable with the workflow, your whole show goes down the tube. A lot of that comfort has to do with the latency involved, and we achieved less than a hundred milliseconds of latency on this past test.”

Using the EVS LSM Connect system, the replay operator can sit at a station at the Altitude studios and remotely control the EVS in the truck a 1,000 miles away.

The EVS replay operators — using a regular EVS controller, router control, intercom, and XFile control — could be in the same office or in another location. For the purposes of the test, the EVS operator was at the Altitude offices and served all three networks. Although there was a slight latency, the operator was able to quickly adjust after some initial testing.

“In terms of EVS,” says Grigely, “our [operator] did not even notice the difference [from a normal onsite production]. Even though control was happening hundreds of miles from where the server was located, there was very little latency.”

Graphics and scorebox operators could also be located at any office and remotely control both units just like the EVS.

Crewing and Cost: Same Personnel, Lower Capex
According to MTVG, Cloud Control is not about cutting back on staffing but about changing the location of the crew for safety and the ability to continue working.

Miller says the replay operator is able to adjust to the new Cloud Control remote setup easily.

“One of our big concerns is our crew,” says Miller. “When we get back to whatever ‘normal’ ends up being, we want to avoid losing any positions. Our shows are pretty lean and efficient anyway with the dual-feed model. With this [Cloud Control] model, while the crew would not be traveling anymore, we would be using the exact same amount of crew and could actually use our home crew all the time for the most part.”

In addition to maintaining the same staffing, RSNs see Cloud Control as a way to become more efficient and save costs in the long term — regardless of the current pandemic situation.

“One of the biggest reasons to do this is that it requires less capital investment from a network,” says Grigely. “A lot of people have always asked why the RSNs haven’t adopted an at-home production model sooner, and the answer was that, in our financial analysis, it just wasn’t that much cheaper; we were already being so efficient in the dual-production model that it just wasn’t worth saving a few flights and hotel rooms [for crew members].

An ethernet switch connected Altitude’s two RVON panels together via a specific IP address to the RTS system inside MTVG’s truck

“And,” he continues, “by the time you build out all of these control rooms for 22 regional sports networks and 40-plus teams, that gets awfully expensive. There was never really an ROI or an incentive for us to do that. But, with [Cloud Control], we think we’ve found a good model that doesn’t require significant investment but provides us with the workflow we need.”

Moving forward, MTVG sees Cloud Control as not just a solution for RSNs but a potential new remote workflow for any sports broadcaster looking for more flexibility and enhanced cloud-based workflows.

“What’s great about this is that it doesn’t require significant capital or time commitment,” says MTVG COO Nick Garvin. “It gives our clients a lot of flexibility and control, and it works for national productions as well as regional. Besides distributed personnel, Cloud Control enables the future of distributed hardware, which can use a combination of onsite hardware, public cloud (AWS/Google Cloud), and MTVG’s private cloud.

Looking Ahead: Next Steps for Truck Providers, RSNs
Although Mobile TV Group sees the Cloud Control model as the path to the future of live sports production, Phil Garvin realizes that this shift must be an industry-wide effort if it is to prove successful.

MTVG’s Cloud Control allows broadcasters to access all the resources of the truck remotely with minimal crew on-site.

“It’s absolutely our intention to work with all other truck providers openly on this,” he explains. “We want to have their engineers talk to our engineers so that everybody can do this together. We’re not trying to do this to gain an advantage over our competitors in any way; we want this to share this for the benefit of all the networks.”

Although the Cloud Control model has been deemed a success by the RSNs that participated in the test last week, its deployment for actual shows remains a work in progress as networks work with leagues to develop a workflow that ensures safety while maintaining broadcast quality.

“There’s going to be a process here,” notes Connelly. “We’re going to have to work with the other RSNs, truck providers, and leagues to all get on the same page. One thing I know for sure: once we make this work, we’re not going to go back. With the safety concerns and technology advances, we believe we can use this [model] to take sports broadcasts to the next level. Usually, when some monumental thing happens, like this pandemic, it forces you to come up with better ways of doing things. I think this is truly a better way of doing things and gives us an opportunity to grow our business.”