Understanding Avaya Internet Protocol Server Interface Resets

I cut my teeth on Port Network (PN) outages when I joined Avaya’s Tier-3 backbone support back in 2006. I was assigned to supporting the S8700 series of duplexed Communication Manager (CM) servers just as CM 3 was being released. Back then, the timers were very tight and a large percentage of my trouble tickets were explaining to customers why an IPSI (Internet Protocol Server Interface) reset, which in turn caused a port network outage.

Avaya uses several different heartbeat mechanisms so that devices know if they have lost connectivity. In the case of port networks, which means any IPSI-controlled cabinet (such as a G650), the heartbeat is variously known as a Sanity Checkslot, Socket Sanity, or IPSI Sanity. This TCP heartbeat is sent to every IPSI every second by the active CM-main (and in duplex CM also by the standby CM). So, if you were to have CM-duplex and duplicated IPSIs in each of the maximum of 64-port networks (2 CM*2 IPSI *64 PN) 256 heartbeats would fly through the network each second.

Originally, the IPSI would react if only three consecutive heartbeats went missing. Starting in CM 3.13, the timer was administrable by an Avaya engineer and in CM 5.0 it became administrable by customers on the CM change system-parameters ipserver-interface form. Now the IPSI Socket Sanity Timeout defaults to 15 seconds (values: 3 to 15 seconds). Data from CM substitutes for missing heartbeats.


Frequently, the cause of missing heartbeats is a mismatch between the IPSI being locked to communicate 100 Mbps/full duplex while the Ethernet switch was set to auto-negotiate (resulting in a half-duplex connection), or vice versa. Also, not enabling quality of service (QoS) to give priority to IPSI traffic, or not segregating the IPSI traffic into a separate physical/virtual LANs, frequently caused problems.

Upon detecting the outage, the IPSI assumes it is sick and reacts by performing a warm reset. During the warm reset, stable calls using resources within the PN stay up. But neither new calls can be initiated nor established calls transition to some other state (e.g. hold) for the obvious reason that there is no connection to CM to manage such transactions. The IPSI’s warm reset generally takes only a few seconds.

If it still doesn’t get heartbeats or data from CM, then after a default of 60 seconds (values: 60 to 120 seconds) the IPSI escalates to a cold reset. All calls using resources within the PN are dropped. On the change system-parameters port-networks form, the PN cold reset delay timer can be modified.

Next, based on the No Service Time Out Interval, the IPSI then waits for a default of 5 minutes (values: 2 to 15 minutes). During that time, while the IPSI is waiting for communication from CM-main, the resources within that PN are unavailable. Note that if one heartbeat gets through, perhaps on a flapping WAN circuit, the timer resets and the countdown starts from the beginning. If the No Service Time Out timer expires, the IPSI then attempts to register to a CM-Survivable Core (SC), formerly known as Enterprise Survivable Servers.


Each IPSI manages its own prioritized list of addresses for up to seven CM-SC, plus the CM-Main, which is always first on the list. Actually, it is in how the CM-SCs are configured that determines the server list for the IPSI. And it is the job of the CM-SC to advertise its own values to the IPSIs so that each IPSI can generate the appropriate list of eight server addresses. A customer can have up to 63 CM-SC. Note that IPSIs cannot register to CM-Survivable Remote (formerly known as Local Survivable Processors) servers.

The preference setting (System Preferred/Local Preferred, Local Only) along with a Community Size field and a Priority Score field, determines the server’s priority on IPSI’s lists. How to assign weighting of these values is beyond the scope of this article.


Each server in a CM-Duplex configuration is constantly comparing its health to the other. One statistic among many they compare is how many IPSIs each one can communicate with right now. If the standby server can communicate with more IPSIs than the active server, the standby takes over and makes itself Active. This can cause frequent server interchanges if an unreliable WAN link connection to a PN causes some of the heartbeats to get lost. So, Avaya introduced the option to Ignore Connectivity in Server Arbitration on the change ipserver-interface n form, thereby potentially reducing interchanges.


I have ignored duplicated IPSIs because I am not a big fan of them. Most of the IPSI-related tickets I’ve received were caused by network issues that duplicated IPSIs would not have protected against.

Based on my experiences, I recommend helping calls in progress stay up as long as they can by delaying the Port Network Cold Reset to 120 seconds. Then I suggest hurrying the registration to a CM-SC by setting the No Service Time Out to 2 minutes.

Although PNs are fading from Avaya’s product mix, they are a solid technology representing 30 years of development. Many customers will rely on them for years to come.

Related Articles:

How to Prevent Media Gateway Split Registrations

Back when Avaya Aura Communication Manager 5.2 was released, I recall reading about this new capability called Split Registration Prevention Feature (SRPF). Although I studied the documentation, it wasn’t until I read Timothy Kaye’s presentation (Session 717: SIP and Business Continuity Considerations: Optimizing Avaya Aura SIP Trunk Configurations Using PE) from the 2014 IAUG convention in Dallas that I fully understood its implications.

What is a Split Registration?

First I need to explain what SRPF is all about. Imagine a fairly large branch office that has two or more H.248 Media Gateways (MG), all within the same Network Region (NR). SRPF only works for MGs within a NR and provides no benefit to MGs assigned to different NRs.

Further, imagine that the MGs provide slightly different services. For example, one MG might provide local trunks to the PSTN, and another might provide Media Module connections to analog phones. For this discussion, it does not matter what type of phones (i.e. SIP, H.323, BRI, DCP, or Analog) exist within this Network Region. During a “sunny day,” all the MGs are registered to Processor Ethernet in the CM-Main, which is in a different NR somewhere else in the network. It aids understanding if you believe that all the resources needed for calls within a NR are provided by equipment within that NR.

A “rainy day” is when CM-Main becomes unavailable, perhaps due to a power outage. When a MG’s Primary Search Timer expires, it will start working down the list trying to register with any CM configured on the Media Gateway Controller (MGC) list. All MGs should have been configured to register to the same CM-Survivable server, which by virtue of their registration to it causes CM-Survivable to become active.

Image 1

In this context a CM server is “active” if it controls one or MGs. A more technical definition is that a CM becomes “active” when it controls DSP resources, which only happens if a MG, Port Network (PN) or Avaya Aura Media Server (AAMS) registers to the CM server.

Since all the MGs are registered to the same CM, all resources (e.g. trunks, announcements, etc.) are available to all calls. In effect, the “rainy day” system behaves the same as the “sunny day” with the exception of which CM is performing the call processing. Even if power is restored, only the CM-Survivable is active, and because no MGs are registered to CM-Main it is inactive.

In CM 5.2, SPRF was originally designed to work with splits between CM-Main and Survivable Remote (fka Local Survivable Processor) servers. In CM 6, the feature was extended to work with Survivable Core (fka Enterprise Survivable Servers) servers. To treat the two servers interchangeably, I use the generalized term “CM-Survivable.”

A “Split Registration” is where within a Network Region some of the MGs are registered to CM-Main and some are registered to a CM-Survivable. In this case only some of the resources are available to some of the phones. Specifically, the resources provided by the MGs registered to CM-Main are not available to phones controlled by CM-Survivable, and vice versa. In my example above, it is likely some of the phones within the branch office would not have access to the local trunks.

Further, the Avaya Session Managers (ASM) would discover CM-Survivable is active. They would learn of CM-Survivable server’s new status when either ASM or CM sent a SIP OPTIONS request to the other. The ASMs then might begin inappropriately routing calls to both CM-Main and CM-Survivable. Consequently, a split registration is even more disruptive than the simple failover to a survivable CM.

What can cause split registrations? One scenario is when the “rainy day” is caused by a partial network failure. In this case some MGs, but not all, maintain their connectivity with CM-Main while the others register to CM-Survivable. Another scenario could be that all MGs failover to CM-Survivable, but then after connectivity to CM-Main has been restored some of the MGs are reset. Those MGs would then register to CM-Main.

How SRPF Functions

If the Split Registration Prevention Feature is enabled, effectively what CM-Main does is to un-register and/or reject registrations by all MGs in the NRs that have registered to CM-Survivable. In other words, it pushes the MGs to register to CM-Survivable. Thus, there is no longer a split registration.

When I learned that, my first question was how does CM-Main know that MGs have registered to CM-Survivable? The answer is that all CM-Survivable servers are constantly trying to register with CM-Main. If a CM-Survivable server is processing calls, then when it registers to CM-Main it announces that it is active. Thus, once connectivity to CM-Main is restored, CM-Main learns which CM-survivable servers are active. This is an important requirement. If CM-Main and CM-Survivable cannot communicate with each other a split registration could still occur.

My second question was how CM forces the MGs back to the CM-Survivable. What I learned was that CM-Main looks up all the NRs for which that Survivable server is administered. The list is administered under the IP network region’s “BACKUP SERVERS” heading. CM-Main then disables the NRs registered to CM-Survivable. That both blocks new registrations and terminates existing registrations of MGs and H.323 endpoints.

Image 2

Once the network issues have been fixed, with SRPF there are only manual ways to force MGs and H.323 endpoints to failback to CM-Main. One fix would be to log into CM-Survivable and disable the NRs. Another would be to disable PROCR on CM-Survivable. An even better solution is to reboot the CM-Survivable server because then you don’t have to remember to come back to it in order to enable NRs and/or PROCR.

Implications of SRPF

Enabling SRPF has some big implications to an enterprise’s survivability design. The first limitation is that within an NR the MGC of all MGs must be limited to two entries. The first entry is Processor Ethernet of CM-Main, and the second the PE of a particular CM-Survivable. In other words, for any NR there can only be one survivable server.

Similarly, all H.323 phones within the NR must be similarly configured with an Alternate Gatekeeper List (AGL) of just one CM-Survivable. The endpoints get that list from the NR’s “Backup Servers” list (pictured above). This also means the administrator must ensure that for each NR all the MGs’ controller lists match the endpoints’ AGL.

Almost always, if SRPF is enabled, Media Gateway Recovery Rules should not be used. However in some configurations enabling both might be desirable. In this case, all MGs must be using an mg-recovery rule with the “Migrate H.248 MG to primary:” field set to “immediately” when the “Minimum time of network stability” is met (default is 3 minutes). Be very careful when enabling both features because there is a danger that in certain circumstances both the SRPF and Recovery Rule will effectively negate each other.

Finally, SPRF only works with H.248 MGs. Port Networks (PN) do not have a recovery mechanism like SRPF to assist in rogue PN behavior.

Enabling SRPF

The Split Registration Prevention Feature (Force Phones and Gateways to Active Survivable Servers?) is enabled globally on the CM form: change system-parameters ip-options.

Image 3

If I had not found Tim Kaye’s presentation, I would not have completely understood SRPF. So, now whenever I come across a presentation or document authored by him, I pay very close attention. He always provides insightful information.

Say Hello to Zang

For months, the Avaya and Esna teams have been hard at work on a revolutionary solution we believe will shape the future of communications in the new digital business era. Last week, the solution, and the new company behind it, were officially unveiled onstage at Enterprise Connect. Say hello to Zang.

We’re incredibly proud of Zang, and are big believers in its potential. Here’s why.

Apps, APIs and SDKs have fundamentally changed the way we connect with one another. Smaller startups have launched freemium, single-feature applications that are gaining traction in the market. And increasingly, we’re meeting small- and midsize customers who are mobile-first and cloud-only.

Zang is our answer to the needs and communication trends we’re seeing in the market.

Zang is the first all-in-one, 100 percent cloud-based communications platform and communication applications-as-a-service. The robust platform gives developers APIs and tools to easily build standalone communication applications, or embed parts of Zang into other apps, online services and workflows.

Zang is virtual, so it’s accessible anywhere, on any device. We also offer a range of ready-to-use applications on a pay-as-you-go subscription basis.

Giving companies the flexibility to build exactly what they need is incredibly powerful.

Imagine a midsize startup and the sheer number of distinct communication and collaboration tools it uses on a daily basis—Gmail for email, Google Docs for document collaboration, Slack for group chat, Salesforce for CRM, Skype for video calls, Zendesk for customer service and smartphones for business communications. Individual teams inside the company may adopt their own subset of tools—Zoho, Hipchat, Google Hangouts, etc.

A lot of important context gets locked up inside each platform, and isn’t shared as employees switch back and forth, all day long, communicating with customers and one another. If you want to embed Slack inside Salesforce, or embed Skype inside Salesforce, it’s hard. These applications are largely single-featured, and aren’t built with easy interoperability at their core.

Zang is different—our platform is more like a set of building blocks. We give companies exactly the configuration they need: either an already-built model, or the pieces needed to add voice, IM, conferencing, video, IVR, or a range of other capabilities to the software they’re already working with. And you don’t have to be a developer or have deep technical expertise to use Zang.

Embedding Zang communication capabilities inside an app takes hours and days, rather than weeks or months like our competitors. Stay tuned, as this is just the beginning of what Zang can do. To learn more and sign up for our First Look Program, please visit Zang.io.

The Sun Was Shining on Avaya in Orlando – With a Nice Cool “Breeze”!

Most people traveling to Orlando go to Disney World for cool rides or a peek at the future in the latest Star Wars adventure. This week, Disney, and other executives from globally-recognized brands, came to visit Avaya instead—to learn more about the company’s future at Enterprise Connect 2016. Put on some sunglasses and see why our future is so bright!

Each year, thousands of communications industry professionals come to the conference to get a preview of the latest innovations in cloud applications, developer tools, and the latest, greatest things they can use to better serve their customers.

Avaya was center stage at the event with a lot of news for 2016 and beyond–a remarkable step in our digital transformation away from the desk phones of the past.

Avaya has a very clever event strategy of inviting customers with their account executives to the exclusive, invitation-only Innovation Lounge: A private room where customers get a peek into the future of communications under NDA.

No, they didn’t see Star Wars VIII, but they did see more than a dozen previews of upcoming Avaya solutions, including customer, team engagement and networking solutions. No cameras allowed and no tweeting, but lots of great sharing of ideas over a drink or two, with customers including Disney, Fidelity Investments, All State, CVS, and hundreds more joined by leading industry analysts from Gartner and Forrester Research.

The lounge is intentionally larger than our show floor booth. It’s money well spent to have your most important prospective customers responsible for purchase decisions hear about the future in a private setting, while still having the show floor presence for media and general attendees.

I had the distinct pleasure of showing the brand-new Avaya Snapp Store, launched Monday morning in conjunction with Avaya Breeze, the next generation of the Engagement Development Platform.

The Snapp Store is an online marketplace for evaluating, purchasing, and downloading software from Avaya and several partners, which can easily “snap in” to Avaya Breeze applications. It’s like having an app store for business software, enabling application developers to obtain software from Avaya and our partners and even from other customers.

Developers can have fully functional, enterprise-grade applications ready in hours and days, not weeks and months. Check it out for yourself. I also shared a preview of an upcoming customer engagement client application for the web, along with a full software development kit. We’ll have more about that soon as we get closer to launch.

But the real star of the show was a new product we kept under wraps until Gary E. Barnett’s keynote address Wednesday morning. Gary began by talking about the incredible potential of Avaya Breeze and the Avaya Snapp Store, with the help of special guests—notably one customer who has already used Breeze to build 40 applications. Yes, 40! The big surprise was the announcement of Zang, a new software and services company from Avaya that will be led by Esna founder Mohammad Nezarati.

This cloud-based communications- and applications-as-a-service platform offers customers ready-to-use apps, and a powerful set of tools to either build their own Zang apps, or embed Zang’s communication capabilities into existing apps, devices and business processes. The interest from attendees was over-the-top positive.

After the keynote, I gave a presentation on the show floor called “20 Cool Applications in 20 Minutes,” to a packed audience. They loved how Breeze and Zang allow customers to design, build  and run communication-enabled business applications with ease.

The last 18 months for me has been extraordinary. Customer engagement. Team engagement. Snapp Store. Avaya Breeze. Zang. Definitely cool.

It’s all part of the transformation of being an innovation-driven software and services company that is more like Google than the IRS. It’s a good time to carry the torch forward with our reputation for earning the business of more than 95 percent of Fortune 100 and 500 businesses – further complemented with an exciting portfolio of new software and services set to meet the needs of people living the digital transformation every day.

Zang translates to the word “excellent,” which is exactly the word used by customers, partners, analysts and attendees to describe Avaya this week.

That feels about as good as the sunshine outside of the Gaylord Palms Resort shining on Avaya!