What is REALLY behind an E911 'Glitch'?
This Blog is also available as an MP3 Audio Podcast.
If your device does not support Flash you can DOWNLOAD THE MP3 HERE
From time to time, people report ‘glitches’ in the E911 system that cause a call to be miss-routed, and ultimately some less than desirable results. One of the more famous incidents was the Washington Post article that read, “Man Found Dead in Office 10 Hours After 911 Phone Glitch Confuses Rescuers”. But was it really a phone ‘Glitch’, or was there something else to blame?
What exactly is a ‘glitch’?
It appears that there is general consensus that the term ‘glitch’ is “A minor malfunction, mishap, or technical problem”, or a “A false or spurious electronic signal caused by a brief, unwanted surge of electric power” .
Astronaut John Glenn is often credited with having the first recorded appearance of the word in 1962 where he wrote “. . . . Another term we adopted to describe some of our problems was ‘glitch'” referring to the various technical problems experienced early on in the space program.
This past month in Muskegon County Oklahoma, there were reports of glitches affecting 911 calls in the area where callers were miss routed to 911 centers hundreds of miles away. The Muskogee Phoenix reported on January 28, 2013 that problems in the E 911 network had surfaced where some calls were going to incorrect towns. Allegedly, and incident took place where a store with IP phones had called 911, however their calls were miss routed costing precious minutes in response time. As in many of these stories, the details surrounding the incident were sketchy and unclear. Was it a “over-the-top” voice over IP telephone service such as Vonage? Or, was it an IP telephone connected back to a corporate PBX in another area? In either case, the blame for this incident was placed on the technology. Unfortunately, most likely it was not the technology that was at fault. Most likely, the culprit was the provisioning of that technology, and the lack of testing.
As corporations flatten consolidate and extend (FCE) there network topologies, processing 911 calls from remote locations can become problematic, especially if those remote locations reside in a remote 911 jurisdiction or service area. Today’s legacy 911 networks are very geographic in nature. Typically they service a very specific area such as a County or consortium of counties. In many cases, the geographic 911 service area boundaries follow the old LATA boundaries established by the telephone company in the 60s and 70s. As states upgrade their E 911 backbone networks, often 911 routing can be statewide.
One of the problems for enterprise users, is that there is no single database source or online reference where one can determine what their 911 service area is. Even if one existed, with the constant flux of network grooming, enterprise administrators don’t have the resources to dedicate keeping up with that changing data.
To solve this problem, Voice over IP Positioning Carriers, or VPC’s, have emerged to provide a more universal “umbrella coverage” in the form of a nationwide 911 network. Although these services do provide a solution for FCE environments, they are not by any means a “set it and forget it” solution. This is where understanding how a specific service works is critical for an IT administrator implementing a hosted E 911 solution.
With most VPC providers, calls are routed from your PBX via SIP trunking, or via a 10 digit number, also known as a PSTN push. Routing at the VPC is done by examining the caller ID, and comparing that to pre-provisioned routing tables in the VPC database. As a failsafe in the network, most VPC services will offer some form of manual emergency call routing service should the number fail translations. “That happens when the provider hasn’t programmed the location in when the phone system is set up,” said Darryl Maggard, Supervisor for Muskogee County E911.
Apparently, this is what happened to the caller in Haskell. The national call center, whose operators are charged with directing calls to the proper E911 center, redirected the call to Muscogee County in Georgia and not Oklahoma.
“That could have been bad,” Maggard said. “And it’s a problem anyone with Voice Over IP service could have and not know about.” He also said customers with Voice Over IP service should call their carrier or their vendor and have them check to ensure emergency calls are routed correctly “Just to be safe”.
Nicholas Maier, Senior Vice President at RedSky Technologies stated, “The incident in question is one where a VoIP service provider has provisioned a customer in their system and has not properly provisioned the location address of the end user. Standard operating procedure when using a VPC is that the location address of the end user is submitted and validated as an MSAG valid address. Once the address is MSAG validated, the call will be routed to the correct PSAP.”
So the ‘glitch’, in this case, was simply someone not following procedure and making sure database information was correct. Meyer added, “There are hundreds of small VoIP service providers in the USA. Not all of them follow best practices”.
This is where great deal of your 911 solution is wrapped around the procedural side of the effort that ensures not only is proper data provided to the network, but the data is checked to make sure it is valid and correct. For example, the address of Avaya in Basking Ridge is 211 Mount Airy Rd. That is a valid address, it is validated in the Master Street Address Guide (MSAG) database, but if I happen to be using that address when I’m on a phone located 47 miles away in my hometown of Ringwood New Jersey, it is completely incorrect and irrelevant to 911.
Karina Yandell, Vice President of Business Development at 911 ETC, Inc. stated “Seconds and minutes matter when a 911 call is placed. We trust that 911 will bring immediate help, but problems exist in our nation’s emergency response system that can cause serious delay in a life or death situation. Your voice provider may or may not have taken the steps necessary to provision location information for outgoing 911 calls. Your employer may or may not have ensured that a 911 call placed from behind the PBX won’t send responders on a wild goose chase.”
As more and more enterprises look at 911 provisioning in their networks, it is not always the technology that is the most critical piece. Proper planning, implementation, and regularly scheduled testing are all key components for an effective 911 solution.
“Tragedies have occurred due to these glitches so it’s imperative for organizations to be proactive about this. The solution is often so simple and affordable that there truly is no excuse for a business or Vo IP provider to not take preventative measures.” Yandell added.
Just like any problem, the first place to start in solving your E911 remediation is a definition of the problem. Immediately follow that with a design session that involves specific use cases around your employees. Once this is established you can easily categorize your issues, define a remediation plan for each scenario or type of user, and establish an implementation schedule. Each group of users in your network have specific needs for their E 911 reporting. When you define what the solution looks like, don’t be afraid to deploy different solutions for different classes of users. E 911 is not difficult, but it requires proper planning. Many times that planning is very unique for your exact environment. Don’t be afraid to hire a consultant, but if you do, be sure to check the references and credentials. It could save your life.
Want more on E9-1-1? E9-1-1 Talk Podcast
Subscribe to my weekly E9-1-1 Talk Podcast here
Thanks for stopping by and reading the Avaya CONNECTED Blog on E9-1-1, I value your opinions, so please feel free to comment below or if you prefer, you can email me privately.
Public comments, suggestions, corrections and loose change is all graciously accepted 😉
Until next week. . . dial carefully.
Be sure to follow me on Twitter @Fletch911