Riot Games Deploys Longest At-Home Production to Date at LoL Mid-Season Invitational in Asia

The esports team produced the event out of its L.A. studios

Few organizations in the live-production business are pushing the envelope in at-home production the way Riot Games’ esports team does. Building on groundbreaking at-home shows from the 2018 League of Legends World Championship and Mid-Season Invitational, Riot Games upped the ante during last month’s 2019 Mid-Season Invitational (MSI), which took place in Vietnam and Taiwan.

The LoL Mid-Season Invitational Final was held at Heping Basketball Gymnasium in Taipei, Taiwan.

“This is the longest remote production we have ever done and definitely one of the longest anyone has ever done,” says Mitch Rosenthal, head of production and operations, esports, Riot Games. “This was by far the most challenging remote production that we’ve ever done as an organization, hands down. But, at the end of the day, viewers at home had no idea, and players at the event had a great experience. It took a lot of big brain power — both internally and externally — to make it happen, but, going forward, there’s not a place in the world that I feel like we can’t [produce a live event].”

In addition to the connectivity challenges presented in a less developed nation like Vietnam, Riot also faced issues in obtaining the infrastructure and equipment needed to pull off one of its largest productions of the year during the Play-In Stage at GG Stadium in Ho Chi Minh City and Group Stage at National Convention Center in Hanoi.

“From a production and operations standpoint, we ran into some unexpected issues in Vietnam, but our team was able to adapt on the fly and figure out some amazing workarounds,” says Rosenthal. “In the end, nobody knew any different except for us, which is one hell of an accomplishment.”

Ho Chi Minh City: Overcoming Transmission, Onsite Challenges
The esports team leverages the company’s Riot Direct network (the ISP used to service all LoL gameplay) to efficiently send feeds from tournament remotes to its Los Angeles production facility. To use Riot Direct for MSI, the esports team was forced to devise a roundabout transmission path: from Los Angeles to Japan to Singapore (where Riot Direct PoP is located) and finally to Vietnam. This more than doubled the latency of Riot’s standard threshold for at-home productions from 180 ms to 300 ms. Riot had three 1-Gbps circuits connecting L.A. to its operations in Ho Chi Minh City and Hanoi.

G2 Esports came away victorious in the LoL Mid-Season Invitational, defeating Liquid 3-0 in the best-of-five Final.

Although the Riot team expected new obstacles, its first large event in Ho Chi Minh City proved especially challenging. The team was unexpectedly unable to deliver its normal equipment kit to the city and, as a result, was forced to create a makeshift production/transmission flypack comprising equipment from the less sophisticated internal system used by the LoL Vietnam Championship Series (VCS) operation.

“We really had to get creative and use every single resource at our disposal to put together a kit that would work for us in Ho Chi Minh,” says Matt Donovan, broadcast engineering manager, Riot Games. “It was completely different from our normal standards for an international show, so it was an incredibly heavy lift to make sure that we were able to deliver that same experience to the players and viewers. But I think the overall broadcast result for fans and the in-game experience for the players were great. Players were happy, and viewers were happy, and, when it comes down to it, that’s all that matters.”

The team was able to cobble together a flypack based on VCS’s Blackmagic Design 60×60 router and ATEM 2M/E switcher. Riot had six transmit lines to L.A. (it typically would have 30 for an international event) and two receive lines from the venue. The production team used the ATEM switcher to subswitch the feeds, and the Los Angeles-based TD subrouted the ATEM switcher over a touchscreen connected to the switcher frame in Ho Chi Minh City.

“The TD was essentially precutting 28 cameras/sources down these six lines from the venue, then cutting between the six subswitched sources in the L.A. control room for air,” says Donovan. “This creative solution ended up being an efficient way to have the same venue I/O the producers were used to without having an equivalent number of transmission paths.”

According to Riot Games Technical Broadcast Manager Michael Caal, a makeshift communications system was created using Unity and Discord to connect L.A. to Ho Chi Minh City using iPhones and laptops. In addition, multiviewer was eventually encoded to provide the directorial team back home a reliable view of all the cameras and allow them to dynamically choose what to be subswitched down the line. Other lines also served as dynamic routes enabling the EVS team back home to grab feeds needed to build highlights packages.

In addition, Riot IT teams had to find sources for more than 80 PCs and network gear that did not arrive. To run the HCM terminal and game servers, they borrowed the gameplay network stack that player teams were going to use to practice and train at their hotel in Hanoi. Riot Production Management was able to secure practice PCs from a local gaming company to outfit the hotel rooms for the pro teams to practice on

G2 Esports walked away with the MSI Trophy this year in Taipei.

“I was really impressed with what the local vendors and the [VCS] facility teams were able to pull off, with the small amount of resources that they had,” says Donovan. “It was definitely a challenge, and it was a great display of collaboration and ingenuity between multiple production and logistics teams. But I think the end viewer and the players got the same experience they would have [if we had our full kit]. To pull that off was pretty spectacular from the production-management side, so I was extremely impressed with how it worked out.”

Further complicating matters, Vietnam’s former President and General Le Duc Anh died on April 25, just before the MSI began. As a result, the Vietnamese government declared two days of mourning on May 3-4, with no public entertainment events to be held. As a result, Riot lost two full show days and had to reallocate the six games scheduled for each day over the course of the rest of the tournament, creating extremely long production days for the crew.

Over the past two years, Riot has collaborated with Haivision to devise an H.264 compression scheme capable of transmitting 40+ 1080p60 video paths using 1 Gbps or less (with a latency of under 300 ms) while synchronizing audio and video within a single frame. Donovan believes that the at-home production workflow in Vietnam wouldn’t have even been possible had Riot not already developed this bleeding-edge workflow.

Evolving At-Home-Production
The current at-home solution has been deployed since last year and was engineered to overcome two of Riot’s former at-home system’s technical constraints: bandwidth and circuit reliability. This MSI event was the first real-world opportunity for Riot to produce a show inside the primary limitation, bandwidth. With only single-gigabit point-to-point links available at each of the three MSI venues (instead of the usual 10-Gb lines required in the past), Riot used Haivision’s low-latency H.264 encoding TS over UDP to produce the same 40 1080p60 video channels and 256 audio channels over just one of the three available 1-Gbps lines. This allowed other venue and production internet traffic to traverse the second line and leave Riot one line for broadcast failover.

The MSI stage was covered in LED, creating a forced perspective that enabled 3D effects and interaction with live talent on stage for the Opening Ceremony.

“With this successful four-week, three-venue, 1-Gbps at-home production under our belts,” says Donovan, “I am curious when we will get the chance to put our tech to the second REMI-limiting constraint that our system was built for: circuit reliability.”

During initial development of the Haivision at-home–production design, Riot engineers and tech teams bench-tested video streams though a homegrown WAN emulator to assess the packet reliability of Haivision’s SRT protocol. Testing results were very positive with reliable video received without large amounts of jitter and packet loss.

“The trade-off here is latency, lots of it,” says Donovan. “Upwards of two seconds of latency would likely be required to utilize the TCP-esque SRT protocol required if Riot were ever to produce a REMI show over lower-priority circuits. Currently, there is no mandate to utilize a venue with limited network reliability, but I would be lying if I said the Riot teams would not be excited to be given the challenge.”

Hanoi: Balancing Resources, Continuing the At-Home Push
After a week of Play-In rounds in Ho Chi Minh City, it was off to Hanoi for the Group Stage, for which Riot had already been loading in throughout the operation in Ho Chi Minh City.

“When we were in Ho Chi Minh City, we were already loading in Hanoi, so we couldn’t have all of our equipment and encoders in Ho Chi Minh City,” Donovan explains. “On international shows like this, we are always trying to find a balance and make the decisions necessary to deliver the highest-quality value with the hardware that we have available.”

Riot continued the at-home workflow in Hanoi, with the bulk of the production centralized in a control room at its L.A. NALCS studios. All switching, camera shading, and graphics insertion and the observer room were located in L.A., along with casters and analysts.

“The way I look at it, we’ve now got the home-run–production [workflow] nailed. Moving forward, it’s more about the degree of difficulty in the regions that we go to around the world,” says Rosenthal. “Connectivity and infrastructure on the ground are two of the major challenges, as well as finding talent locally instead of having to fly people in.”

Taipei: LED-Fueled Show, At-Home Workflow
Riot Games’ LoL International Finals productions are always marked by grandeur, and this year’s MSI Final at Heping Basketball Gymnasium in Taipei, Taiwan, continued that tradition. Spearheaded by Riot’s CBLOL production team out of Brazil, the design covered the stage in LED, creating a forced perspective that enabled 3D effects and interaction with live talent on stage.

To allow space for the Opening Ceremony and also serve gameplay, Riot had to develop new player sleds that could easily move on and off the stage. At the same time, it had to engineer a way to put equipment that normally lives in a rack backstage inside the sled while minimizing space, power, and heat – all while maintaining strict power and network redundancy.

“The umbilicals that run to the sled were minimal, consisting of power, Ethernet, and fiber,” Caal explains. “In the end, we deployed a 3-phase, 32-A, 220-V [for power], a handful of Ethernet cables for LED, and two Tac-12 fibers for video and network transport, making the sled cabling easier to move and manage in the few minutes it had allocated to be repositioned during the show.”

The at-home innovation continued in Taipei, where Riot Games produced an Opening Ceremony remotely for the first time. Although nearly the entire production of last year’s World Championship was an at-home production, a flypack onsite was used for the ultra-complex, tech-heavy Opening Ceremony. In Taipei, although Riot rolled out a small 4K truck to aid in the production, the entire feed was switched in L.A.

“In the beginning, we were extremely vendor-dependent for just running the at-home shows. We have evolved a lot, to the point where we are now focused not only on [using at-home] efficiency and sustainability but also consistently pushing the envelope,” says Rosenthal. “One of our manifestos is not to do what the industry says is standard just because it’s the standard. We’re going to figure out what works best and put that on the forefront — regardless of what everyone else is doing. I really loved seeing that from all the areas of the group during MSI.”