As Dove Valley Undergoes Facelift, Denver Broncos Ramp Up IT Architecture
The Denver Broncos headquarters at Dove Valley in Englewood, CO, is undergoing a massive $35 million makeover this summer that will include a sparkling new 115,000-square-foot facility and a 12,000-square-foot expansion of the team’s current facility. However, that’s not the only enhancement taking place in Dove Valley, as the Broncos’ IT team continues to build out its IT and storage infrastructure for its football operations and creative services teams.
“For the most part, the football offseason has been IT’s on-season this year,” says Denver Broncos VP of IT Russ Trainor. “We try to get all the upgrades in, all the new systems online, and everything set to have it all up and running by the time training camp starts [in late July].”
Four Key Pillars to Orange-and-Blue IT Success
The Broncos IT architecture is built on four key pillars: NetApp storage, Cisco networking with Nexus switches at the core, VMware cloud-virtualization software, and Cisco UCS (Unified Computing System) servers. This infrastructure drives the team’s entire football operations (including the PlayerLinc playbook iPad app), creative services (in-venue video board show and digital-media video production), and administrative departments.
“Over the last two years we put a lot of focus on those four core pillars and we have seen a big difference,” says Trainor. “We ripped out our entire network and put a new Cisco network in with Nexus at the core. And we probably only have one or two servers now that are not certified [through] VMware. Everything else has gone VMware. We love it. Who wouldn’t love that kind of flexible technology.”
Like Looking in a Mirror: Dove Valley and Sports Authority Field
In building out their system, Trainor and company were set upon installing uniform platforms at both Sports Authority Field (its primary data center) and Dove Valley (secondary data center) to make it easier to manage content between the two sites, as well as provide asynchronous disaster recovery. With that in mind, the Broncos worked with Denver-based systems integrator PEAK Resources to install a NetApp FAS2240-4 storage system at both facilities. Dove Valley and Mile High are connected and mirrored via a 100-Mbps Metro-Ethernet connection (though Trainor hopes to bump this up to 1-Gbps pipe in the near future and 10GbE connectivity at some point down the line).
“We definitely want our system on the same platforms across both those sites to make it easier on us for managing,” says Trainor. “We wanted to make our storage systems the same, so we went out and got the two NetApp units last year to serve as our primary production system. Those are for our VMware, all our servers, user-shares, and do all of our backup over the wire at night.”
This past spring, the Broncos brought in an additional NetApp FAS2240-4 system specifically for the football operations’ video needs. Each FAS2240-4 boasts a 10-Gbps connection back to the Nexus core, so the Broncos have an aggregate 20-Gbps connecting each controller to the core.
“The 2240-4 is a great fit, performance wise and that makes it easier to manage since we don’t have a big IT shop,” says Trainor. “The 2240 [storage systems] are all running the latest and same flavor [software], so that way, when we go to manage it, we can log in and see all systems and they all look the same from the controller side.”
A NetApp-centric Shop
In addition, the Broncos Creative Services team, which oversees the Sports Authority Field game-day video-board presentation, utilizes a FAS2040 (a holdover from the team’s prior infrastructure) for its second-tier storage at the Stadium – totaling four NetApp systems in all.
“We do more all the time with the NetApp system on the backend. So even on the football side, more and more items are falling onto the NetApp side of things,” says Trainor. “Wherever we can leverage their infrastructure and platform, we try to take advantage of it.”
The stadium and training facilities’ wired and wireless network infrastructures are built on Cisco Nexus and 3750 series switches, Cisco 3500 and 3600 series access points, and Cisco 5508 and 2504 controllers – all of which are integrated with the NetApp storage.
The same NetApp infrastructure is also integrated with variety of other core aspects of organization, including the Microsoft Exchange Server email-system, accounting software, and the ThirdLight digital-asset-management (DAM) server that is utilized to access event photos, remote-user servers and file-shares, event management applications, security systems, and various applications servers.
Trainor and his team utilize a wide palate of NetApp OnCommand management software features to control the storage systems as a single pool of resources across the SAN environment, including OnCommand Balance (monitoring and optimizing the performance of virtual servers); OnCommand Protection Manager, SnapMirror, and SnapVault (to protect data through replication between data centers over the wire); and Snapshot, SnapRestore , the Virtual Storage Console, and SnapManager (for Microsoft Exchange and Microsoft SQL Server protect data locally).
“We leaned on NetApp a lot,” says Trainor. “We run our current backups between sites and so we use NetApp Protection Manager for that, then we use Snapshot and SnapMirror from one site to the other so you can get a nice DR [disaster recovery] and business continuity infrastructure overlaid on top of it, which is always a good thing. It sure beats running tapes back and forth.”
NetApp de-duplication also helps optimize the storage environment by significantly reducing storage requirements.
“Some of the de-dup pieces from the NetApp side has been a big win for us in terms of saving dollars in storage,” adds Trainor.
PlayerLinc Boosts Playbook Efficiency, Saves Trees
This NetApp storage environment, in conjunction with VMware and Cisco elements, allowed the IT department to work with the Broncos’ video team to develop a PlayerLync digital-playbook iPad application. The app, which can be accessed on any iPad via WiFi, provides game-film video playback, updated playbooks, and a variety of communication features to Broncos coaches, players, and personnel execs. The secure data resides on virtual servers running on NetApp, eliminating the need for hard-copy playbooks.
“Our tablet [PlayerLync application] all runs off of a VMware PlayerLinc server,” says Trainor. “That is all in-house and running on the backend NetApp [infrastructure]. From our perspective, it’s been a big win-win for both the players and us, as well as for PlayerLinc.”
In addition to the football operations and PlayerLinc, the system is responsible for storing and feeding the video-production operations for DenverBroncos.com and the Team Media department.
“All the backend elements for [DenverBroncos.com] content creation is also us,” says Trainor. “Those guys always want more terabytes because they’re always producing new [content]. All the editing and production is done in-house. They store it and share it on the backend platform, which we use NetApp for.”
FlexPod on the Way
Next up on the horizon for Trainor will be the integration of FlexPod technology, which combines NetApp storage systems, Cisco UCS servers, and Cisco Nexus fabric into a single, flexible architecture.
“The next big item coming up for us in the next month or two is the FlexPod piece and getting that going,” says Trainor. “We are hoping to do FlexPod at the stadium. It is basically an integration of all our core technologies together to provide us with [better performance]. We will get that in by training camp. If that all goes well then we will do the same configuration here at Dove Valley.”