How This Billion Dollar Mall Operator Virtualized its Unified Communications System

Forest City Enterprises is a massive real estate management firm based in Cleveland, Ohio. With nearly $11 billion in property under management, Forest City’s best-known properties include the Westfield San Francisco Centre mall in downtown San Francisco and the Atlantic Yards mixed-use redevelopment project, which includes the new Barclays Center basketball arena where the Brooklyn Nets, play.
 
Forest City reaps more than $1 billion in revenue a year and employs more than 2,500 people nationwide. Despite its size, the company’s workers were largely content with its old phone system – a stable first-generation VoIP-based system for internal calls and PRI and T1 connections for outside ones, says Bradley J. Coyne, telecom specialist for Forest City, who candidly shared his experience in a talk at the International Avaya Users Group’s Converge 2013 conference in Orlando this week.
The bread-and-butter system was 7 years old. While it was not loaded with features, it boasted Five 9s (99.999%) reliability, says Coyne – the most important factor in an old-school business like real estate. “If the telephones are down, we are closed for business,” he said. 
Nevertheless, Forest City was looking to upgrade its communications system and add SIP underneath in order to gain full Unified Communications capabilities, as well as consolidate its networks. The organization was already two months into the upgrade to the latest version of Avaya Aura when its system integrator, NACR, approached them with an enticing proposal: besides upgrading to Avaya Aura, why not also virtualize Aura, in order to gain efficiencies, better uptime and disaster recovery, and scalability?
 
Despite being potentially NACR’s first customer running Aura on VMware, Coyne and his compatriots in IT were enthusiastic. Forest City was no stranger to VMware. Over the past 5 years, Forest City had virtualized 80% of its servers using VMware – virtually all (pun intended) of its non-communication servers. That had allowed Forest City to boost the number of applications it ran (it now had 500 virtual machines) while slashing the number of physical servers it maintained from 150 to 50.
Slide4.JPG
Slides courtesy of Bradley J. Coyne of Forest City Enterprises.

Convincing Forest City’s top brass took some effort, mostly because of worries that this last-minute change would delay the new Aura rollout, says Coyne. But he and his team were able to show that rolling out Aura on VMware would be worth any delays.
More difficult, ironically, was for Coyne and his communications team to build relationships with the server infrastructure team it never had had close dealings with – and upon whom they would suddenly have to rely heavily. Post virtualization, they would have the power to wipe out servers controlling phone routing and user data information for thousands of employees with the incorrect push of a button. Not only were processes created to prevent that scenario from happening, but new relationships and a spirit of teamwork between formerly siloed departments within IT were created as a result of this upgrade, says Coyne.
 
Slide6.JPG
Through diligent work, NACR and Forest City were able to get several weeks ahead of the original schedule. But the switchover to the new virtualized communications systems did have some scary moments due, ironically, to miscommunication. 
During the weekend cutover of the main phone system, Forest City realized that it had told NACR’s engineers that it had already signed up a SIP carrier. In fact, its phones were still using the H.323 protocol. Moving forward could’ve resulted in the erasure of the routing data and phone numbers of thousands of deskphones. 
For a moment, Coyne thought, “Uh oh, we might not have jobs next Monday.”
Fortunately, Forest City and NACR were able to roll back to backups of the old phone servers before continuing on its upgrade. “We dodged a huge bullet,” he said. When Monday morning rolled around, no employees called to complain. Only the employees’ phones’ call logs were missing. “It was such a smooth cutover that users couldn’t tell that their entire phone system had changed. When they asked what we’d done, we’d reply, ‘Oh, nothing big.'”
Similarly, the changeover of the voice mail system to the new Avaya Aura Messaging also resulted in minimal complaints from users, who only noted that the touch tones sounded different.
From 22 physical communications servers, Forest City now runs 43 virtual and physical server instances. Besides running far more than twice the number of applications than before, these servers are more secure, easier to recover and move around in case of server failure or disaster, and easier to manage. For instance, firmware updates to the virtual servers take just 5 minutes, versus 28-30 minutes as before.
These will all combine to save Forest City’s IT big-time, predicts Coyne, especially when the company completes its SIP trunk upgrades.
With Avaya Aura 6.2, Forest City’s many traveling workers will also begin to be able to use softphones and communication apps such as Avaya Flare Communicator and one-X Communicator on their laptops and mobile devices. Coyne isn’t rushing with those upgrades, though, as he wants to make them as painless as possible for the workers. He plans to use SAP Afaria to roll out the apps to mobile devices.
While Forest City uses LifeSize for video conferencing today, it is looking into replacing that with Avaya Scopia and Aura Web video conferencing, says Coyne. He and his team are also working on turning on other Aura features such as faxing over IP.
“Every week we install another application or turn on another feature, all without impacting our users,” Coyne said. “We’re getting gold stars in IT from the business units, which is really gratifying.”