Activision Blizzard Esports Blazes Cloud-Based Remote-Production Trail With Grass Valley’s New GV AMPP Platform

Esports giant helped develop GV’s new SaaS production platform and is first to deploy it live

In April 2018, Activision Blizzard Esports knew that its newly minted Overwatch League operation would be moving from centralized tournaments at Blizzard Arena in Burbank, CA, to a home-and-away model in 2020. With that in mind, the broadcast team was on the lookout for a fully cloud-based remote-production system, which did not yet exist, to serve Blizzard Esports’ ultra-customized live-production needs. Enter Grass Valley.

The GV AMPP feature set can be fully customized on a show-to-show basis.

“At NAB 2018, we sat down with Grass Valley and told them we were going to need totally new technology and unique solutions when [Overwatch League] goes home-and-away, so what can we do together?” recounts Pete Emminger, VP, global broadcast, Activision Blizzard Esports. “Internally, we had this project focused on cloud production that we were calling Project Snowstorm, which eventually became Project 2020. And Grass Valley was looking to launch this suite of cloud production services that eventually became AMPP. We knew we needed a system along those lines because we have such an integrated workflow, so it made a lot of sense to work together.”

In the ensuing months, Blizzard and Grass Valley teamed up to develop what would become GV AMPP (Agile Media Processing Platform), a cloud-based SaaS production solution that allows users to spin resources up or down on an as-needed basis.

With its esports leagues forced to shift from in-venue events to online tournaments due to the coronavirus pandemic, Blizzard is using AMPP as its virtual control room and master-control solution to deliver live Overwatch League (OWL) and Call of Duty League (CDL) events to fans across the globe — with the entire crew located safely at their respective homes. 

Accelerated Timeline: Coronavirus Makes a 2021 Initiative a 2020 Reality
During last summer’s OWL home stands, Blizzard’s broadcast team was covertly running AMPP in parallel to the onsite truck production and remotely switching the show from its Irvine, CA, facility via the cloud. In February, AMPP went live as Overwatch League’s master-control solution at the start of the 2020 season and was scheduled to be rolled out for live event production in 2021. Then came the coronavirus pandemic.

A Call of Duty League at-home–production workstation

“When the coronavirus situation began canceling the events in China, we started to work on a [remote-production] solution immediately,” says Emminger. “Then our internal tech team and the Grass Valley team came up with the idea of just using AMPP, which was already heavily in development. We were planning on leaving AMPP as a master-control solution for the rest of the season and then migrating to production usage in 2021, but, when our [events] moved to online gameplay, it completely blew up that timeline, and we decided to move everything to AMPP right away.”

Blizzard’s technology team — led by Corey Smith, director, live operations, Global Broadcast — worked almost round-the-clock with Grass Valley to boost the audio capabilities and deploy additional toolsets within the AMPP platform that would be needed for a live-production environment.

“Our tech team and Grass Valley were able to complete a massive tech lift to get it fully stood up in a very short time,” says Emminger. “I’m extremely proud of the entire team, and it was truly a team effort.”

How AMPP Works: Metered Microservices Create Ultra-Flexible Platform
With the AMPP Master Control application, Blizzard can create highly configurable virtual master-control rooms accessible via web-based interface from anywhere in the world — without the need for a conventional equipment room.

Activision Blizzard Esports’ entire broadcast team is located at their respective homes and producing OWL and CDL broadcasts via GV AMPP in the cloud.

“We are now 100% in the cloud; there is no physical infrastructure whatsoever in terms of switchers or audio consoles,” Emminger explains. “The only physical hardware is people’s computers, cameras, microphones, and a comms panel. Other than that, there’s absolutely no physical infrastructure required.”

The fully browser-based and cloud-agnostic AMPP platform allows multiple distribution streams with separate graphics and languages for each audience region to be produced by a single operator. All monitoring and local program-distribution processes take place entirely in the cloud.

“We saw that we needed to be able to deliver on the flexibility and the agility that cloud-based systems can afford because broadcasters have a lot of needs that are very elastic in nature,” says Mike Cronk, VP, core technology, Grass Valley. “You may need one thing for one show and but not for another, and the cloud is perfect for that. So we set off to build a cloud-based system that, over time, could encompass the entire Grass Valley product line. And Blizzard has been a key partner almost from the outset.”

AMPP also allows production teams to flexibly create multiple workflows, with dedicated nodes supporting multiviewers, test-signal generators, switching, audio mixing, and recording. Users can rapidly and easily add, remove, change, or redistribute media-production and delivery features with AMPP depending on needs.

“When you put together an AMPP production, you pick from a list of [tools] for what you need in your broadcast,” says Smith. “That’s pretty amazing because you don’t typically get to go into a control room and say, ‘I need A, B, and C for this week’s show, and then, next week, I’m going to need X, Y, and Z’ without buying a bunch of hardware or renting gear. In the AMPP world, we basically have a catalog of services that we can just fire up when we need them for each show. If we want to make something else a part of the show, we can just add an element with the click of a button.”

AMPP is built on a microservices architecture based on five core technologies: fabric, timing, connectivity, identity, and streaming. In terms of cost structure, AMPP essentially functions as a series of metered microservices, and each tool (switcher, audio mixer, multiviewer, clip player) has a different metered rate associated with it. Therefore, the user is charged only for the features activated and the amount of time each feature is active.

“Since we’re a very hungry company, we want to break the boundaries of technology, and we’re focused on how we can create things differently,” says Ryan Cole, senior tech manager, Global Broadcast. “Right from the beginning, we challenged Grass in a lot of different ways and identified specific things that we would need. We worked closely with them to produce something pretty epic that we’re using right now.” 

The Production: Inside Blizzard’s AMPP-Fueled Operation
Blizzard is running several instances in which AMPP serves as a virtual control room for the different pieces of its live-production ecosystem. The main match production and Watchpoint pre/postgame studio show each have their own AMPP fabric, both feeding into the master-control fabric. The master-control layer packages the final program feed, which is distributed out to YouTube (Blizzard’s exclusive streaming partner), linear-TV outlets, and rightsholders around the world.

The OWL GV AMPP setup features (clockwise from top left) a virtual audio mixer, switcher, multiviewer, program preview/playout, clip player, and on-air feed.

In addition, the six-person observer team (in-game virtual camera operators), comprising five observers and an in-game director, has its own submix AMPP to create the in-game action. Each observer is tied into a custom GV Engine software-based processing engine, which allows the observer to operate in an ultra-low–latency environment. The mixed in-game feed is sent as a single source into the main-match–production AMPP switcher.

Each instance of AMPP can be fully customized, depending on the toolset necessary for the applications. The match-production fabric, for example, features a virtual audio mixer, switcher, multiviewer, program preview/playout, clip player, and on-air return feed.

“Essentially, AMPP is the traditional production workflow melded into a single online platform view,” says Smith. “AMPP basically works as a set of microservices running in the background. So, every time there is a button pressed or an action that needs to be taken by the system, it engages the various microservices in the background to make all of this happen.”

Blizzard has integrated a Singular cloud-based graphics system. It is also using vMix cloud-based instant-replay software to create replays and then using the clip player within AMPP for video playback. Since AMPP is fully API-based, Smith says, additional third-party production tools will be incorporated in the future.

Blizzard has been using GV AMPP for Overwatch League master control since the start of the 2020 season.

After initially relaunching its broadcasts without player-cams, Blizzard has integrated these POV camera feeds into its CDL coverage, and OWL is soon to follow. These webcams are controlled by a simple Raspberry Pi developer board, ingested into the AMPP platform, and integrated into the live stream. Showcasing players in their at-home environments not only brings a sense of intimacy to the telecast but also allows fans to see players’ backdrops and jerseys, which are key from a sponsorship perspective.

“If you look at the production value of the online show we are [producing] in the cloud today, it is basically the same level as the in-venue show,” says Smith. “I think that’s pretty impressive compared to what most broadcasters and esports [outlets] are putting on the air right now, which is mostly cut-down shows just to get back on the air. We’re still doing the same quality show; the only difference is, our trucks are now in the cloud and our crew and casters are sequestered in their bedrooms and living rooms.”

The Crew: The Usual Suspects Aside From a One-Man Front Bench
In terms of crew, Blizzard’s OWL and CDL at-home productions feature largely the same positions as for an onsite production in the truck — with one major difference: a single producer essentially serves as the entire front bench, taking on the roles of director, producer, and TD as well as monitoring the audio levels within the AMPP interface.

Overwatch League caster Soe Gschwind enjoys her lunch while announcing live OWL action from home.

“I think [the front bench] is where you’re going to see the biggest shift as we move into more cloud-based production,” says Emminger. “I think it’s going to resemble the way lighting designers and directors follow along themselves and don’t take quite as many hard cues. I think we’re going to eventually see more of that from a front-bench perspective.”

In addition to this front-bench producer, each OWL and CDL also features one or two graphics operators, one or two replay ops, an op for the clip player (to play back music or video clips), and the six-person observer team (five observers and one in-game director), as well as casters/talent. Casters were provided with custom equipment kits, including a 4K camera, lighting, and a backdrop.

Blizzard provided casters with custom equipment kits, including a 4K camera, lighting, and a backdrop.

“From a crewing perspective, we’ve learned that you pretty much need almost all the same positions,” says Emminger. “A lot of people assume that moving online means you need less, but it actually winds up being similar. That’s for the same reason you would have a position on a regular show: if you want good-quality replay, you need somebody to focus and watch for good action. Just [being] online doesn’t change the need for quality people.”

In all, Emminger estimates, roughly 60 people are involved in the overall cloud-based operation, including casters/talent, observers, production, and technical crew. Currently, Blizzard is using AMPP for OWL matches in both the North America and Asia (the league has teams in South Korea and China), which means that the operation runs nearly around the clock.

“It was a pretty major challenge to get everybody trained,” says Smith. “Because we already had it in use [for master control], it made it a bit simpler. This was definitely a huge change for the entire crew — both for production and for talent. Everybody had to learn new production workflows and new tech: AMPP and TeamSpeak comms plus new graphics and replay technologies.”

The Comms: Gaming-Focused TeamSpeak Keeps Crew in Touch
Blizzard is relying on TeamSpeak VoIP technology for all comms, including broadcast operations, casters/talent, game operations/referees, and for the teams’ comms during gameplay. Developed specifically for the gaming community, the ultra-low-latency comms software has been adapted to serve more broadcast-centric workflows. For example, Blizzard sets up subgroup channels for each part of the production (for example, main production, observer room, WatchPoint pre/postgame show) the same way that a traditional broadcast would typically set up partylines and PLs for comms.

An Overwatch League at-home-production workstation

“It has been a great unified workflow across the entire operation,” says Cole. “TeamSpeak is similar to other comms platforms, but it has a really interesting feature that allows us to do point-to-point and partyline comms. The idea is, you can basically set up hotkeys to whisper to specific individuals or groups of people. This allows us to do a lot of different features, not just talking point to point but also talking in between channels, between different talent, talkback, and IFB — all on the platform itself.”

To adapt to the remote-production model, Blizzard provided each crew member and caster with an Elgato Stream Deck and interfaced it with TeamSpeak hotkeys. That way, users can have a hardware-based comms panel like they are used to working within the TeamSpeak software.

Since TeamSpeak is being used for both the broadcast operation and gameplay, Blizzard has also been integrating audio from teams’ comms for CDL into the broadcast over the top of gameplay video, creating a unique behind-the-scenes experience for viewers.

The Future: Go Back to Truck or Stay in AMPP? Likely a Mix of Both
Although all Activision Blizzard Esports events will continue to be online-only for the foreseeable future, live in-venue events will be back at some point. With a fully cloud-based architecture now in place, the question becomes, Will these productions return to the truck or remain in the cloud?

Hammerstein Ballroom in NYC was one of two hosts (along with Esports Stadium Arlington in Texas) for Overwatch League Opening Weekend in February.

“In the immediate future, if and when we got back to a LAN-based gameplay in a studio, I think we will probably keep using AMPP except with our talent and commenters back onsite at the desk,” says Emminger. “That’s our current feeling, but we’re working on making those decisions now.”

However, he adds that there are still limitations to AMPP — especially in terms of capacity for larger productions — and Grass Valley will have to keep building up the platform with more-robust audio tools, as well as further integration with third-party production systems.

“There are some limitations in AMPP for things like number of inputs, so it wouldn’t work for a show as big as our OWL Grand Final show anytime soon,” adds Emminger. “AMPP also doesn’t have the capability to tie into our other in-venue systems like LED screen control, and we need to be heavily integrated into those types of things. AMPP is fully API-based, so those elements will be integrated, but we don’t have all those workflows built out yet.”

While the future of live production remains to be seen, Emminger, Smith, Cole, and their team are just excited to be back delivering live esports to fans.

“Obviously, the COVID situation is a really difficult reality for everyone right now, so for us to be able to get content out there and have some fun with our community and keep them involved has been amazing,” says Cole. “You’ll see in our broadcasts right now how we’re actively engaging our community and trying to have more fun with them, and they’re doing the same thing with us. We’re just happy to be able to be able to serve the fans.”

Grass Valley will present more details about its launch of GV AMPP during an online event Thursday, April 23 at 11 a.m. ET. Interested viewers can register for GV LIVE Presents — Innovate 2020 using this registration link: https://grass-valley.brand.live/gv-live-presents.