What is REALLY behind an E911 'Glitch'?

This Blog is also available as an MP3 Audio Podcast.
If your device does not support Flash you can DOWNLOAD THE MP3 HERE

From time to time, people report ‘glitches’ in the E911 system that cause a call to be miss-routed, and ultimately some less than desirable results. One of the more famous incidents was the Washington Post article that read, “Man Found Dead in Office 10 Hours After 911 Phone Glitch Confuses Rescuers”. But was it really a phone ‘Glitch’, or was there something else to blame?

What exactly is a ‘glitch’?
It appears that there is general consensus that the term ‘glitch’ is “A minor malfunction, mishap, or technical problem”, or a “A false or spurious electronic signal caused by a brief, unwanted surge of electric power” .

Astronaut John Glenn is often credited with having the first recorded appearance of the word in 1962 where he wrote “. . . . Another term we adopted to describe some of our problems was ‘glitch'” referring to the various technical problems experienced early on in the space program.

This past month in Muskegon County Oklahoma, there were reports of glitches affecting 911 calls in the area where callers were miss routed to 911 centers hundreds of miles away. The Muskogee Phoenix reported on January 28, 2013 that problems in the E 911 network had surfaced where some calls were going to incorrect towns. Allegedly, and incident took place where a store with IP phones had called 911, however their calls were miss routed costing precious minutes in response time. As in many of these stories, the details surrounding the incident were sketchy and unclear. Was it a “over-the-top” voice over IP telephone service such as Vonage? Or, was it an IP telephone connected back to a corporate PBX in another area? In either case, the blame for this incident was placed on the technology. Unfortunately, most likely it was not the technology that was at fault. Most likely, the culprit was the provisioning of that technology, and the lack of testing.

As corporations flatten consolidate and extend (FCE) there network topologies, processing 911 calls from remote locations can become problematic, especially if those remote locations reside in a remote 911 jurisdiction or service area. Today’s legacy 911 networks are very geographic in nature. Typically they service a very specific area such as a County or consortium of counties. In many cases, the geographic 911 service area boundaries follow the old LATA boundaries established by the telephone company in the 60s and 70s. As states upgrade their E 911 backbone networks, often 911 routing can be statewide.

One of the problems for enterprise users, is that there is no single database source or online reference where one can determine what their 911 service area is. Even if one existed, with the constant flux of network grooming, enterprise administrators don’t have the resources to dedicate keeping up with that changing data.

To solve this problem, Voice over IP Positioning Carriers, or VPC’s, have emerged to provide a more universal “umbrella coverage” in the form of a nationwide 911 network. Although these services do provide a solution for FCE environments, they are not by any means a “set it and forget it” solution. This is where understanding how a specific service works is critical for an IT administrator implementing a hosted E 911 solution.

With most VPC providers, calls are routed from your PBX via SIP trunking, or via a 10 digit number, also known as a PSTN push. Routing at the VPC is done by examining the caller ID, and comparing that to pre-provisioned routing tables in the VPC database. As a failsafe in the network, most VPC services will offer some form of manual emergency call routing service should the number fail translations. “That happens when the provider hasn’t programmed the location in when the phone system is set up,” said Darryl Maggard, Supervisor for Muskogee County E911.

Apparently, this is what happened to the caller in Haskell. The national call center, whose operators are charged with directing calls to the proper E911 center, redirected the call to Muscogee County in Georgia and not Oklahoma.

“That could have been bad,” Maggard said. “And it’s a problem anyone with Voice Over IP service could have and not know about.” He also said customers with Voice Over IP service should call their carrier or their vendor and have them check to ensure emergency calls are routed correctly “Just to be safe”.

Nicholas Maier, Senior Vice President at RedSky Technologies stated, “The incident in question is one where a VoIP service provider has provisioned a customer in their system and has not properly provisioned the location address of the end user. Standard operating procedure when using a VPC is that the location address of the end user is submitted and validated as an MSAG valid address. Once the address is MSAG validated, the call will be routed to the correct PSAP.”

So the ‘glitch’, in this case, was simply someone not following procedure and making sure database information was correct. Meyer added, “There are hundreds of small VoIP service providers in the USA. Not all of them follow best practices”.

This is where great deal of your 911 solution is wrapped around the procedural side of the effort that ensures not only is proper data provided to the network, but the data is checked to make sure it is valid and correct. For example, the address of Avaya in Basking Ridge is 211 Mount Airy Rd. That is a valid address, it is validated in the Master Street Address Guide (MSAG) database, but if I happen to be using that address when I’m on a phone located 47 miles away in my hometown of Ringwood New Jersey, it is completely incorrect and irrelevant to 911.

Karina Yandell, Vice President of Business Development at 911 ETC, Inc. stated “Seconds and minutes matter when a 911 call is placed. We trust that 911 will bring immediate help, but problems exist in our nation’s emergency response system that can cause serious delay in a life or death situation. Your voice provider may or may not have taken the steps necessary to provision location information for outgoing 911 calls. Your employer may or may not have ensured that a 911 call placed from behind the PBX won’t send responders on a wild goose chase.”

As more and more enterprises look at 911 provisioning in their networks, it is not always the technology that is the most critical piece. Proper planning, implementation, and regularly scheduled testing are all key components for an effective 911 solution.

“Tragedies have occurred due to these glitches so it’s imperative for organizations to be proactive about this. The solution is often so simple and affordable that there truly is no excuse for a business or Vo IP provider to not take preventative measures.” Yandell added.

Just like any problem, the first place to start in solving your E911 remediation is a definition of the problem. Immediately follow that with a design session that involves specific use cases around your employees. Once this is established you can easily categorize your issues, define a remediation plan for each scenario or type of user, and establish an implementation schedule. Each group of users in your network have specific needs for their E 911 reporting. When you define what the solution looks like, don’t be afraid to deploy different solutions for different classes of users. E 911 is not difficult, but it requires proper planning. Many times that planning is very unique for your exact environment. Don’t be afraid to hire a consultant, but if you do, be sure to check the references and credentials. It could save your life.

Want more on E9-1-1?  E9-1-1 Talk Podcast
Subscribe to my weekly E9-1-1 Talk Podcast here

Thanks for stopping by and reading the Avaya CONNECTED Blog on E9-1-1, I value your opinions, so please feel free to comment below or if you prefer, you can email me privately.

Public comments, suggestions, corrections and loose change is all graciously accepted 😉
Until next week. . . dial carefully.

Be sure to follow me on Twitter @Fletch911


Related Articles:

Avaya Named a Leader in Gartner’s Magic Quadrant for Contact Center Infrastructure

Avaya is honored to be recognized as a leader in Gartner’s Magic Quadrant for Contact Center Infrastructure worldwide. Avaya has been the only vendor having the distinction of being named a Leader for 16 consecutive years. Each year the research organization creates a market view of key players for business users, reflecting business goals, needs, and priorities.

Contact centers have gone beyond phone calls with customers now expecting to communicate on their terms via text, IM, email, chat or video. For the past 16 years Avaya has created seamless and highly personalized experiences, building brand loyalty for companies all around the world.

According to Deloitte, 85% of organizations view customer experience provided through contact centers as a competitive differentiator. Todays companies must remain relevant by creating a single interface to connect customers with the correct resource each time, supporting their preferences. Supervisors and managers need real-time performance information to adapt immediately to situations to ensure optimized customer experience.

Avaya has focused its efforts on creating next-generation contact center solutions, creating communication strategies enabling a continuous transition between channels during customer interactions.

Please visit Gartner’s page to read the full report and see how Avaya’s Contact Center infrastructure continues to deliver best-of-breed Contact Center applications. We look forward to continuing innovation and leading business communications for the digital age.


Seeing into the Office of the Future

Dubai is heavily focused on delivering on its Smart City goals, with the goal of being among the smartest—and happiest—cities in the world. The drive toward smart cities is part of a wider shift, with countries around the globe seeing a migration from rural areas to urban. With more than half the world’s population now living in cities, organizations in the Middle East are facing increasingly difficult decisions about how they allocate resources and manage their workforce.

For a city like Dubai, that can be challenging. Finding the right real-estate location for office space, managing energy usage and providing physical workspaces for employees working different shifts in a modern, 24X7 city creates its fair share of headaches. Enterprises also have to cope with an increasing Bring-Your-Own-Device (BYOD) culture, with smartphone and device penetration especially high in the United Arab Emirates and other Gulf countries.

We have reached an inflection point where the number of devices connected and generating data is accelerating at an exponential level. Our work environments are beginning to blur, as workspaces are no longer physical but virtual. Organizations need to deliver a consistent work experience whether in an office or on the road, or at home. The key driver is to intuitively connect not only communications devices, but interact with the ambient technologies that surround us, like lighting, BMS and signage.

At Avaya, we believe that the Smart Enterprise is one of the key building blocks for smart cities—and one of the foundations of true Smart Enterprise development will be what we term the Office of the Future. This is about more than just technology; the Office of the Future involves automating work processes within the enterprise to deliver a more intuitive employee experience. As with any “smart” solution, the Office of the Future is only smart if it resolves the challenges it aims to address.

So what do we mean by Office of the Future? Imagine walking into a new office you’ve never visited before and your work station is ready before you sit down, configured to your preferences, right down to the air conditioning settings and digital signage displayed with your identity. When a client arrives to meet you at the office, they can be sent to a meeting room automatically, and you can talk to them on their mobile as you walk to greet them.

With Avaya solutions, the act of an employee booking a workspace would kick start a workflow that immediately sets up that space with all communication devices enabled and connected securely, while interacting with the building management system to ensure the environment was set to the employee’s requirements, everything from lighting to temperature to digital signage. The employee could be identified by their phone or a card ID. When they leave, the workspace can be reset for the next employee that reserves it.

Such Smart workspaces will help enterprises manage their resources more effectively, better leverage their real estate investments, and improve employee well-being and productivity. Here in Dubai, the Office of the Future starts now.

89% of Employees Apparently Don’t Care About Mobile Security

Mobile Security Avaya

IT security has a big job: keep corporate data safe in the face of motivated hackers and unaware employees. Today that job is harder than ever — employees are bringing their own devices and applications into the office every morning, and walking out the door with corporate data every night.

The intention behind Bring Your Own Device and Bring Your Own Apps is good: Employees want to be productive away from the office. No one wants to carry around two smartphones, or truck around two laptops while they’re on the road. Cloud-based work apps excel at document version control, are accessible everywhere, and help teams cut down on email as a collaboration tool.

The reality of BYOD and BYOA is more troublesome: If your company is one of the estimated 26 percent with no official BYOD policy in place, employees will load work email and work documents on their personal mobile devices anyway. If a company fails to give their employees the cloud-based apps they want, they’ll simply use the app’s consumer-grade version. Thousands of unsecured laptops and smartphones get lost or stolen every week. It’s estimated that 43 percent of U.S. companies have experienced a data breach in the last year alone.

Troubling numbers

Given that backdrop, ask yourself — how many mobile devices are out there with your company’s data on them? The answer might surprise you.

In a recent survey of more than 12,000 people, security software maker Kaspersky Lab found roughly half used personal smartphones, tablets or laptops for work, 36 percent kept work files on their personal devices, 34 percent accessed work-related email from personal devices, and somewhere between 11 to 18 percent carried around corporate passwords.

Asked about it, just 11 percent said they were seriously concerned about keeping work-related information secure on their personal mobile devices.

If your company doesn’t have formal policies in place around personal mobile devices, chances are, your corporate data is already heading home with employees each night. BYOD and BYOA are just the start— Bring Your Own Everything is on the horizon.

Embracing the present

The first step is to either build a BYOD and BYOA policy, or review your existing policies to keep them up-to-date.

Employees are already using their own devices and apps inside the workplace — in an April 2015 report, Netskope found the average organization is now using 730 cloud-based applications. If that number seems high, it may be time to audit the software your teams are using, and determine if sensitive corporate information is at risk of being lost in the cloud.

Next, give employees the secure tools they need to use the devices and apps they choose. Different teams may choose different engagement software based on their individual needs. Mandating the entire company standardize on a single, monolithic software platform or official device is unrealistic, and may encourage “shadow IT,” where teams ignore official channels and adopt the tools that work for them.

Information silos are dangerous. At best, silos hinder company engagement by preventing teams from getting the information they need to make informed decisions easily. At worst, silos force employees to kluge together a solution — for example, emailing data across the company in spreadsheets.

Breaking information silos apart is possible with software like the Avaya Engagement Development Platform, which allows companies to write custom code that either communication-enables their existing apps, or builds new apps to share data between silos.

Lastly, smart companies adopt multiple layers of security, knowing that data breaches are just as likely to come from within the company than without.  Firewalls are not enough — network access must be segmented and role-based.

In a widely-publicized data breach last year, a major U.S. retailer admitted it had lost millions of consumer credit card numbers after it gave its HVAC vendor access to wide swaths of the company’s corporate network. Hackers breached the vendor, and used their network credentials to raid the retailer’s credit card database, which was sitting in a section of the network that an HVAC company should not have been able to access.

Virtualized, software-defined networking makes role-based network access easy, reduces the size of the network’s footprint of endpoints and obscures portions of the network from hackers. Individual devices, applications and endpoints are provisioned dynamically, with network access extending and retracting as needed.

BYOD and BYOA represent the new reality for enterprises. Take proactive steps to review your company’s BYOD and BYOA policies, give employees the tools to allow this trend, share information securely between applications and gain more control over the corporate network.

Want more? Download the new Avaya white paper, “The New Rules of Engagement.”