September 25, 2021 3:30pm – We are getting reports of our customers not being able to receive calls from Verizon. Right now, only Verizon has been tested. We’re working on other tests to see if we can narrow down the issue.
We will post updates here as we have them.
Update 3:40pm – We have confirmed that this is a Verizon issue. It is affecting Verizon wireless customers and customers of cellular services which use their networks such as Spectrum and Straight Talk. Downdetector is also reporting Verizon issues here: https://downdetector.com/status/verizon/.
Update 3:55pm – It looks like the worst has passed. Downdetector is showing that the reports of issues are starting to drop and we have had reports of Verizon calls going through to our customers now.
September 27, 2021 – 10:00am – We are not sure the issues are related to this weekend’s events, but our carrier has posted a bulletin stating this:
“NOTICE: Currently a large underlying carrier is experiencing intermittent service disruption. This outage is causing inbound calls of some numbers to not be delivered to SIPTRUNK. We apologize for any service interruption that these issues may cause. “
Update 11:30am – We have discovered that DTMF tones are no longer working on digital receptionists. DTMF tones are the tone made by your phone when you dial on the keypad and they are used to choose options on the menus in phone systems (for instance, push 1 for sales, push 2 for support, etc.). Because of this, people are not able to make choices on those menus. Some customers have their systems set to hang up if no choice is made to weed out telemarketers. Right now, those systems may be weeding out everyone. If this seems to be the case on your system, please contact us and we can change the system so that “no choice made” goes somewhere. We have alerted our carrier to the issue in case they are unaware.
Update 11:48am – This seems to be a widespread issue as we have now received a report from a customer using AT&T as the carrier who is having similar issues.
Update 12:00pm from SIPTRUNK – “Our carrier is experiencing an intermittent issue with inbound and outbound calls. This includes audio & DTMF tones. But static is usually only introduced at analog equipment, so we suggest rebooting any physical devices you might have, including router, modems, firewalls, PBXs, etc. But we have multiple tickets open with the carrier. All teams are actively engaged at this time. Currently no ETA. We will provide an update as soon as one is available.”
Update 3:45pm – SIPTRUNK reports that the issue is still under investigation.
Update 6:35pm – Our carrier reports that the network behavior has returned to normal. We are changing phone system programming that was changed because of the DTMF issues. Please let us know if you are still experiencing abnormal phone system behavior.
Update September 28 8:45am – Although the carrier reported that things were back to normal, we are still receiving reports or issues (and experiencing them ourselves). We have reported this and are awaiting news.
Update 4:30pm – Updates have been slow to come today. Our carrier has given us a bit more information now though. They are seeing about 1/7 the tickets today as yesterday, so things are improving. The issue is not a misconfiguration, but rather is a DDOS attack directed at VOIP carriers. The affected carriers have not released that it is a ransomware attack, but that happened recently to a large Canadian carrier last week and the rumor is that this is related and may be the same attackers. Earlier, Verizon was the suspected source of the issue and even Verizon themselves thought they were the problem. However, the underlying carrier currently being attacked is Bandwidth. Because of the interconnections between companies, it seemed to be a Verizon problem at first because they had the most end users affected. Bandwidth is a large “behind the scenes” carrier. You can check their Down Detector information here: https://downdetector.com/status/bandwidth/ and their own status page here: https://status.bandwidth.com/. We expect them to post “everything normal” later today as off-hours voice traffic drops a lot and reduces the impact of the attacks, but we may see the same issues return tomorrow if they have not been able to mitigate the attacks.
Update September 29 8:45am – Currently, it looks like a repeat of yesterday, where daytime business traffic is starting to have an impact on performance of the network. Downdetector is showing an increase of reported issues again: https://downdetector.com/status/bandwidth/. Our carrier is working on diversifying their routes to other upstream carriers, but this involves a lot of manual porting of hundreds of thousands of numbers, so this will take some time. Mitigation of the attack at the carrier level is improving, so we expect less problems as time goes on, but there *are* still issues happening.
Update 10:45am – Carrier has posted an issue this morning but are currently saying it only affects inbound calls from the Verizon network. Of course, other providers using the Verizon network such as Spectrum Mobile, Straight Talk, etc. will also be affected. Here is another recent news article with a little more information: https://www.zdnet.com/article/bandwidth-ceo-confirms-outages-caused-by-ddos-attack/
Update 4:00pm – Siptrunk sent out a letter regarding the issue. No real extra news, but some clarifications. Here is an excerpt from the letter we received:
–BEGIN
As you may be aware, one of the US’s largest underlying carriers (ULC) has been under a Distributed Denial-of-Service (DDOS) attack this week. This attack is similar but larger in scope than the attack on VoIP.MS that started on September 16th. Since this is a ULC we can easily route around them on outbound calls, but this is not the case for inbound calls. The ULCs serve as the “carrier of record” for the DIDs or telephone numbers used by our customers. The current design of the US telephone network means that all calls destined to a particular telephone number have to be handled by the carrier of record. The only way to change this is to port or move the number to a new carrier of record and for landlines, this process can take 5 to 7 days under normal circumstances.
SIPTRUNK is investigating all options to address intermittent inbound call issues caused by the DDOS on the ULC, but please know this is affecting large and small customers the likes of which including Microsoft Teams, Google voice, RingCentral, etc… to just name a few.
We have seen significant improvement on the ULC that is being attacked, and we have confidence they will get their network restored to 100% capacity as quickly as possible.
However, with that said SIPTRUNK has been working with other ULCs to arrange porting of DIDs to alternative ULCs. As you can imagine there are a number of carriers and customers attempting mass ports and this is causing congestion in the porting process. We will continue to work to have inbound calls delivered via alternative ULCs.
–END
September 30 – 7:15am – For the technically curious, here is a good video between a well-known MSP (Managed Service Provider) and a VOIP provider on Youtube discussing the issue. It uses a lot of jargon, but may be interesting to some of you. https://www.youtube.com/watch?v=CeCd9zW-R5o. It is from yesterday, so when they say “yesterday”, they are talking about the 28th. However, not much has changed.
Update 1:45pm – Upstream providers report much better stability of the network today. There are still some users reporting issues, but the number and severity seems to be diminished.
Update 4:15pm – SIPTRUNK has released another statement. Here are the relevant parts:
–BEGIN
All SIPTRUNK Network Services remain operational. We continue to monitor the underlying carrier (ULC) that has been impacted over this week. At this time, they remain under a DDoS attack however they are taking the necessary steps to limit service disruptions. We continue to see significant improvements as they fine-tune their DDOS mitigation efforts, and we have confidence they will get their network restored to 100% capacity as quickly as possible.
We continue to collaborate with other ULCs to organize emergency ports from the impaired carrier to their services. It has become clear that the DDoS attacks are impacting the ability of Providers to port numbers. Providers are experiencing significant porting delays and we are also aware of the potential disruptions that could occur when moving numbers. We will continue to test and try alternative solutions to move beyond the control of the attackers.
Please understand this is not just an attack on a ULC but an attack on the United States Infrastructure and it has visibility at all levels of the federal government.
–END
Update October 1 4:15pm – All ULC services have been stabilized and calls are processing normally. We performed a scheduled port today and it went through without issue. We believe this issue has now been resolved.
If you have any questions, please text 256-547-6817 or email support@internetpro.net.