How This Billion Dollar Mall Operator Virtualized its Unified Communications System

Forest City Enterprises is a massive real estate management firm based in Cleveland, Ohio. With nearly $11 billion in property under management, Forest City’s best-known properties include the Westfield San Francisco Centre mall in downtown San Francisco and the Atlantic Yards mixed-use redevelopment project, which includes the new Barclays Center basketball arena where the Brooklyn Nets, play.
Forest City reaps more than $1 billion in revenue a year and employs more than 2,500 people nationwide. Despite its size, the company’s workers were largely content with its old phone system – a stable first-generation VoIP-based system for internal calls and PRI and T1 connections for outside ones, says Bradley J. Coyne, telecom specialist for Forest City, who candidly shared his experience in a talk at the International Avaya Users Group’s Converge 2013 conference in Orlando this week.
The bread-and-butter system was 7 years old. While it was not loaded with features, it boasted Five 9s (99.999%) reliability, says Coyne – the most important factor in an old-school business like real estate. “If the telephones are down, we are closed for business,” he said. 
Nevertheless, Forest City was looking to upgrade its communications system and add SIP underneath in order to gain full Unified Communications capabilities, as well as consolidate its networks. The organization was already two months into the upgrade to the latest version of Avaya Aura when its system integrator, NACR, approached them with an enticing proposal: besides upgrading to Avaya Aura, why not also virtualize Aura, in order to gain efficiencies, better uptime and disaster recovery, and scalability?
Despite being potentially NACR’s first customer running Aura on VMware, Coyne and his compatriots in IT were enthusiastic. Forest City was no stranger to VMware. Over the past 5 years, Forest City had virtualized 80% of its servers using VMware – virtually all (pun intended) of its non-communication servers. That had allowed Forest City to boost the number of applications it ran (it now had 500 virtual machines) while slashing the number of physical servers it maintained from 150 to 50.
Slides courtesy of Bradley J. Coyne of Forest City Enterprises.

Convincing Forest City’s top brass took some effort, mostly because of worries that this last-minute change would delay the new Aura rollout, says Coyne. But he and his team were able to show that rolling out Aura on VMware would be worth any delays.
More difficult, ironically, was for Coyne and his communications team to build relationships with the server infrastructure team it never had had close dealings with – and upon whom they would suddenly have to rely heavily. Post virtualization, they would have the power to wipe out servers controlling phone routing and user data information for thousands of employees with the incorrect push of a button. Not only were processes created to prevent that scenario from happening, but new relationships and a spirit of teamwork between formerly siloed departments within IT were created as a result of this upgrade, says Coyne.
Through diligent work, NACR and Forest City were able to get several weeks ahead of the original schedule. But the switchover to the new virtualized communications systems did have some scary moments due, ironically, to miscommunication. 
During the weekend cutover of the main phone system, Forest City realized that it had told NACR’s engineers that it had already signed up a SIP carrier. In fact, its phones were still using the H.323 protocol. Moving forward could’ve resulted in the erasure of the routing data and phone numbers of thousands of deskphones. 
For a moment, Coyne thought, “Uh oh, we might not have jobs next Monday.”
Fortunately, Forest City and NACR were able to roll back to backups of the old phone servers before continuing on its upgrade. “We dodged a huge bullet,” he said. When Monday morning rolled around, no employees called to complain. Only the employees’ phones’ call logs were missing. “It was such a smooth cutover that users couldn’t tell that their entire phone system had changed. When they asked what we’d done, we’d reply, ‘Oh, nothing big.'”
Similarly, the changeover of the voice mail system to the new Avaya Aura Messaging also resulted in minimal complaints from users, who only noted that the touch tones sounded different.
From 22 physical communications servers, Forest City now runs 43 virtual and physical server instances. Besides running far more than twice the number of applications than before, these servers are more secure, easier to recover and move around in case of server failure or disaster, and easier to manage. For instance, firmware updates to the virtual servers take just 5 minutes, versus 28-30 minutes as before.
These will all combine to save Forest City’s IT big-time, predicts Coyne, especially when the company completes its SIP trunk upgrades.
With Avaya Aura 6.2, Forest City’s many traveling workers will also begin to be able to use softphones and communication apps such as Avaya Flare Communicator and one-X Communicator on their laptops and mobile devices. Coyne isn’t rushing with those upgrades, though, as he wants to make them as painless as possible for the workers. He plans to use SAP Afaria to roll out the apps to mobile devices.
While Forest City uses LifeSize for video conferencing today, it is looking into replacing that with Avaya Scopia and Aura Web video conferencing, says Coyne. He and his team are also working on turning on other Aura features such as faxing over IP.
“Every week we install another application or turn on another feature, all without impacting our users,” Coyne said. “We’re getting gold stars in IT from the business units, which is really gratifying.”

Related Articles:

Zang Serves Up a Special Delivery for Your Mom this Mother’s Day

Mother’s Day is the one day in the U.S. when the most phone calls are made. According to this cool Mother’s Day Facts site, 122 million calls are made to mothers on Mother’s Day in the United States alone. Considering there are only 85 million mothers in the U.S., Mom must be pretty busy taking calls from her multiple children, and Dad must be busy making reservations at the favorite family restaurant (Mother’s Day remains the top holiday for dining out).

To help make sure Mom gets that special call on Mother’s Day, Zang today announced a Zang-built service for those who 1) are multiple time zones away from mom (ie: military, working or studying abroad), 2) just want to send another thoughtful gift to Mom to let her know she’s loved, or 3) frankly, for those who have a track record for forgetting (you know who you are). With the Zang Forget Me Not service, anyone can record a voicemail for their mom before Mother’s Day, designate the date & time the voicemail should be sent, then receive a text confirming the voicemail was delivered. The new service was created using  cloud-based Zang Comms platform as a service, which allows anyone to create communication applications and services just like Forget Me Not.

How does it work, you ask? Simple. First go to and complete four short steps:

1)  Enter your telephone phone number
2)  Enter recipient’s telephone number
3)  Pick the time you would like the recording to be delivered
4)  Zang Forget Me Not service will then call your phone number for you to record, review and approve your message for delivery.


Go ahead—give it a try! It’s just one more surprise you can give Mom this Mother’s Day.

Next time you visit Dubai, take a public transport

With happiness being a key focus in Dubai, government agencies are looking towards contributing to the goal of raising the quality of life of customers and ensuring public happiness. These agencies are quickly realizing that the key to delivering a better and more personalized experience is technology. Using the latest services and solutions paves the way to guaranteed customer retention and loyalty.

One of the leading organizations in the area of customer care, winning multiple awards for its contact centre operations including a Hamdan bin Mohammed Smart Government Award, is the Roads & Transport Authority (RTA).

The RTA has a wide remit including Dubai’s Metro, public buses, private road vehicle registration, traffic management and more, so it has a diverse customer base negotiating Dubai’s busy transport system, with a volume of customer enquiries to match. It therefore comes as no surprise that the RTA is investing in multiple channels of communications with its customers, to improve standards of service, increase efficiency and gain valuable feedback from its user. It is also looking to technology to help improve the quality of interactions with clients and to improve overall levels of customer satisfaction and engagement. It has utilized a number of different solutions to increase its outreach to customers, and over time the focus of these efforts has evolved, to include voice communications, smart apps and multi-channel engagement.

From projects and operational perspective, RTA has a big focus on alternative smart channels. It offers 173 smart services under nine apps, that can help customers complete their transactions with a click of the finger through the automation of the main services the authority provides. It is dedicated to opening up more channels of communication, with an omni-channel strategy, that includes delivering services through channels such as self-service kiosks. At present the RTA has deployed around 16 kiosks, which offer smart services to users in RTA service centres, and in future it plans to have around 100 kiosks all over the city. The Authority has a well-established customer care line, which handles enquiries across the range of its activities, running on Avaya contact centre solutions. In 2015, the centre handled over 2.5 million calls, with over 80% of calls responded to in 20 seconds, and 90% of issues resolved in one call.

To make this possible, last year the contact centre underwent a major technology refresh, to put in place the latest generation of solutions. With Avaya Aura, RTA is now using the most recent software to increase the efficiency of the contact centre. With the aim to deliver the best possible interaction experience to transport customers, Avaya aligned with RTA’s Customer Resource Management strategy to consolidate channels and mediums into RTA’s first, best-in-class contact center to host multi-channel interactions. Among the capabilities that the new technology has enabled is an advanced Interactive Voice Response (IVR) system, which has helped to improve operations by automatically handling some of the more common customer enquiries. On New Year’s Eve the centre received some 12,000 calls, with the IVR handling one third of all enquiries.

The RTA is a pioneering example of how technology can make the difference in delivering quality to customers through the creation of a seamless and hassle free experience. As we share the RTA’s vision in excelling in customer experiences to achieve happiness, my advice to you is that, next time you visit Dubai, remember to take a  public transport.

How to Prevent Media Gateway Split Registrations

Back when Avaya Aura Communication Manager 5.2 was released, I recall reading about this new capability called Split Registration Prevention Feature (SRPF). Although I studied the documentation, it wasn’t until I read Timothy Kaye’s presentation (Session 717: SIP and Business Continuity Considerations: Optimizing Avaya Aura SIP Trunk Configurations Using PE) from the 2014 IAUG convention in Dallas that I fully understood its implications.

What is a Split Registration?

First I need to explain what SRPF is all about. Imagine a fairly large branch office that has two or more H.248 Media Gateways (MG), all within the same Network Region (NR). SRPF only works for MGs within a NR and provides no benefit to MGs assigned to different NRs.

Further, imagine that the MGs provide slightly different services. For example, one MG might provide local trunks to the PSTN, and another might provide Media Module connections to analog phones. For this discussion, it does not matter what type of phones (i.e. SIP, H.323, BRI, DCP, or Analog) exist within this Network Region. During a “sunny day,” all the MGs are registered to Processor Ethernet in the CM-Main, which is in a different NR somewhere else in the network. It aids understanding if you believe that all the resources needed for calls within a NR are provided by equipment within that NR.

A “rainy day” is when CM-Main becomes unavailable, perhaps due to a power outage. When a MG’s Primary Search Timer expires, it will start working down the list trying to register with any CM configured on the Media Gateway Controller (MGC) list. All MGs should have been configured to register to the same CM-Survivable server, which by virtue of their registration to it causes CM-Survivable to become active.

Image 1

In this context a CM server is “active” if it controls one or MGs. A more technical definition is that a CM becomes “active” when it controls DSP resources, which only happens if a MG, Port Network (PN) or Avaya Aura Media Server (AAMS) registers to the CM server.

Since all the MGs are registered to the same CM, all resources (e.g. trunks, announcements, etc.) are available to all calls. In effect, the “rainy day” system behaves the same as the “sunny day” with the exception of which CM is performing the call processing. Even if power is restored, only the CM-Survivable is active, and because no MGs are registered to CM-Main it is inactive.

In CM 5.2, SPRF was originally designed to work with splits between CM-Main and Survivable Remote (fka Local Survivable Processor) servers. In CM 6, the feature was extended to work with Survivable Core (fka Enterprise Survivable Servers) servers. To treat the two servers interchangeably, I use the generalized term “CM-Survivable.”

A “Split Registration” is where within a Network Region some of the MGs are registered to CM-Main and some are registered to a CM-Survivable. In this case only some of the resources are available to some of the phones. Specifically, the resources provided by the MGs registered to CM-Main are not available to phones controlled by CM-Survivable, and vice versa. In my example above, it is likely some of the phones within the branch office would not have access to the local trunks.

Further, the Avaya Session Managers (ASM) would discover CM-Survivable is active. They would learn of CM-Survivable server’s new status when either ASM or CM sent a SIP OPTIONS request to the other. The ASMs then might begin inappropriately routing calls to both CM-Main and CM-Survivable. Consequently, a split registration is even more disruptive than the simple failover to a survivable CM.

What can cause split registrations? One scenario is when the “rainy day” is caused by a partial network failure. In this case some MGs, but not all, maintain their connectivity with CM-Main while the others register to CM-Survivable. Another scenario could be that all MGs failover to CM-Survivable, but then after connectivity to CM-Main has been restored some of the MGs are reset. Those MGs would then register to CM-Main.

How SRPF Functions

If the Split Registration Prevention Feature is enabled, effectively what CM-Main does is to un-register and/or reject registrations by all MGs in the NRs that have registered to CM-Survivable. In other words, it pushes the MGs to register to CM-Survivable. Thus, there is no longer a split registration.

When I learned that, my first question was how does CM-Main know that MGs have registered to CM-Survivable? The answer is that all CM-Survivable servers are constantly trying to register with CM-Main. If a CM-Survivable server is processing calls, then when it registers to CM-Main it announces that it is active. Thus, once connectivity to CM-Main is restored, CM-Main learns which CM-survivable servers are active. This is an important requirement. If CM-Main and CM-Survivable cannot communicate with each other a split registration could still occur.

My second question was how CM forces the MGs back to the CM-Survivable. What I learned was that CM-Main looks up all the NRs for which that Survivable server is administered. The list is administered under the IP network region’s “BACKUP SERVERS” heading. CM-Main then disables the NRs registered to CM-Survivable. That both blocks new registrations and terminates existing registrations of MGs and H.323 endpoints.

Image 2

Once the network issues have been fixed, with SRPF there are only manual ways to force MGs and H.323 endpoints to failback to CM-Main. One fix would be to log into CM-Survivable and disable the NRs. Another would be to disable PROCR on CM-Survivable. An even better solution is to reboot the CM-Survivable server because then you don’t have to remember to come back to it in order to enable NRs and/or PROCR.

Implications of SRPF

Enabling SRPF has some big implications to an enterprise’s survivability design. The first limitation is that within an NR the MGC of all MGs must be limited to two entries. The first entry is Processor Ethernet of CM-Main, and the second the PE of a particular CM-Survivable. In other words, for any NR there can only be one survivable server.

Similarly, all H.323 phones within the NR must be similarly configured with an Alternate Gatekeeper List (AGL) of just one CM-Survivable. The endpoints get that list from the NR’s “Backup Servers” list (pictured above). This also means the administrator must ensure that for each NR all the MGs’ controller lists match the endpoints’ AGL.

Almost always, if SRPF is enabled, Media Gateway Recovery Rules should not be used. However in some configurations enabling both might be desirable. In this case, all MGs must be using an mg-recovery rule with the “Migrate H.248 MG to primary:” field set to “immediately” when the “Minimum time of network stability” is met (default is 3 minutes). Be very careful when enabling both features because there is a danger that in certain circumstances both the SRPF and Recovery Rule will effectively negate each other.

Finally, SPRF only works with H.248 MGs. Port Networks (PN) do not have a recovery mechanism like SRPF to assist in rogue PN behavior.

Enabling SRPF

The Split Registration Prevention Feature (Force Phones and Gateways to Active Survivable Servers?) is enabled globally on the CM form: change system-parameters ip-options.

Image 3

If I had not found Tim Kaye’s presentation, I would not have completely understood SRPF. So, now whenever I come across a presentation or document authored by him, I pay very close attention. He always provides insightful information.