News and Updates

September 27, 2021 – 9:30am – We are experiencing an issue with a shared web server which is affecting some customers.   We are currently investigating the issue as it is taking longer to recover than we initially thought.  We’ll be posting updates here as we get them.

Update 10:03am – Services have been restored.  A combination of DOS (denial of service) attack and hardware failure was responsible.  We are working on a hardware fix.

If you have any questions or concerns about this, please email us at support@internetpro.net, text or call 2565476817, or use the chat box in the lower right of your screen.

September 25, 2021  3:30pm  – We are getting reports of our customers not being able to receive calls from Verizon.  Right now, only Verizon has been tested.   We’re working on other tests to see if we can narrow down the issue.

We will post updates here as we have them.

Update 3:40pm – We have confirmed that this is a Verizon issue.   It is affecting Verizon wireless customers and customers of cellular services which use their networks such as Spectrum and Straight Talk.   Downdetector is also reporting Verizon issues here: https://downdetector.com/status/verizon/.

Update  3:55pm – It looks like the worst has passed.   Downdetector is showing that the reports of issues are starting to drop and we have had reports of Verizon calls going through to our customers now.

September 27, 2021 – 10:00am – We are not sure the issues are related to this weekend’s events, but our carrier has posted a bulletin stating this:
“NOTICE: Currently a large underlying carrier is experiencing intermittent service disruption. This outage is causing inbound calls of some numbers to not be delivered to SIPTRUNK. We apologize for any service interruption that these issues may cause. “

Update 11:30am – We have discovered that DTMF tones are no longer working on digital receptionists.  DTMF tones are the tone made by your phone when you dial on the keypad and they are used to choose options on the menus in phone systems (for instance, push 1 for sales, push 2 for support, etc.).   Because of this, people are not able to make choices on those menus.  Some customers have their systems set to hang up if no choice is made to weed out telemarketers.  Right now, those systems may be weeding out everyone.  If this seems to be the case on your system, please contact us and we can change the system so that “no choice made” goes somewhere. We have alerted our carrier to the issue in case they are unaware.

Update 11:48am – This seems to be a widespread issue as we have now received a report from a customer using AT&T as the carrier who is having similar issues.

Update 12:00pm from SIPTRUNK – “Our carrier is experiencing an intermittent issue with inbound and outbound calls. This includes audio & DTMF tones. But static is usually only introduced at analog equipment, so we suggest rebooting any physical devices you might have, including router, modems, firewalls, PBXs, etc. But we have multiple tickets open with the carrier. All teams are actively engaged at this time. Currently no ETA. We will provide an update as soon as one is available.”

Update 3:45pm – SIPTRUNK reports that the issue is still under investigation.

Update 6:35pm – Our carrier reports that the network behavior has returned to normal.  We are changing phone system programming that was changed because of the DTMF issues.   Please let us know if you are still experiencing abnormal phone system behavior.

Update September 28 8:45am – Although the carrier reported that things were back to normal, we are still receiving reports or issues (and experiencing them ourselves).   We have reported this and are awaiting news.

Update 4:30pm – Updates have been slow to come today.   Our carrier has given us a bit more information now though.  They are seeing about 1/7 the tickets today as yesterday, so things are improving.   The issue is not a misconfiguration, but rather is a DDOS attack directed at VOIP carriers.  The affected carriers have not released that it is a ransomware attack, but that happened recently to a large Canadian carrier last week and the rumor is that this is related and may be the same attackers.   Earlier, Verizon was the suspected source of the issue and even Verizon themselves thought they were the problem.  However, the underlying carrier currently being attacked is Bandwidth. Because of the interconnections between companies, it seemed to be a Verizon problem at first because they had the most end users affected.   Bandwidth is a large “behind the scenes” carrier.   You can check their Down Detector information here: https://downdetector.com/status/bandwidth/   and their own status page here: https://status.bandwidth.com/.   We expect them to post “everything normal” later today as off-hours voice traffic drops a lot and reduces the impact of the attacks, but we may see the same issues return tomorrow if they have not been able to mitigate the attacks.

Update September 29  8:45am – Currently, it looks like a repeat of yesterday, where daytime business traffic is starting to have an impact on performance of the network.  Downdetector is showing an increase of reported issues again: https://downdetector.com/status/bandwidth/.   Our carrier is working on diversifying their routes to other upstream carriers, but this involves a lot of manual porting of hundreds of thousands of numbers, so this will take some time.   Mitigation of the attack at the carrier level is improving, so we expect less problems as time goes on, but there *are* still issues happening.

Update 10:45am – Carrier has posted an issue this morning but are currently saying it only affects inbound calls from the Verizon network.  Of course, other providers using the Verizon network such as Spectrum Mobile, Straight Talk, etc. will also be affected.  Here is another recent news article with a little more information:  https://www.zdnet.com/article/bandwidth-ceo-confirms-outages-caused-by-ddos-attack/

Update 4:00pm – Siptrunk sent out a letter regarding the issue.  No real extra news, but some clarifications.   Here is an excerpt from the letter we received:

–BEGIN

As you may be aware, one of the US’s largest underlying carriers (ULC) has been under a Distributed Denial-of-Service (DDOS) attack this week. This attack is similar but larger in scope than the attack on VoIP.MS that started on September 16th.    Since this is a ULC we can easily route around them on outbound calls, but this is not the case for inbound calls. The ULCs serve as the “carrier of record” for the DIDs or telephone numbers used by our customers. The current design of the US telephone network means that all calls destined to a particular telephone number have to be handled by the carrier of record. The only way to change this is to port or move the number to a new carrier of record and for landlines, this process can take 5 to 7 days under normal circumstances.

SIPTRUNK is investigating all options to address intermittent inbound call issues caused by the DDOS on the ULC, but please know this is affecting large and small customers the likes of which including Microsoft Teams, Google voice, RingCentral, etc… to just name a few.

We have seen significant improvement on the ULC that is being attacked, and we have confidence they will get their network restored to 100% capacity as quickly as possible.

However, with that said SIPTRUNK has been working with other ULCs to arrange porting of DIDs to alternative ULCs. As you can imagine there are a number of carriers and customers attempting mass ports and this is causing congestion in the porting process. We will continue to work to have inbound calls delivered via alternative ULCs.

–END

September 30 – 7:15am – For the technically curious, here is a good video between a well-known MSP (Managed Service Provider) and a VOIP provider on Youtube discussing the issue.  It uses a lot of jargon, but may be interesting to some of you.  https://www.youtube.com/watch?v=CeCd9zW-R5o.  It is from yesterday, so when they say “yesterday”, they are talking about the 28th.  However, not much has changed.

Update 1:45pm – Upstream providers report much better stability of the network today.  There are still some users reporting issues, but the number and severity seems to be diminished.

Update 4:15pm – SIPTRUNK has released another statement.  Here are the relevant parts:
–BEGIN
All SIPTRUNK Network Services remain operational. We continue to monitor the underlying carrier (ULC) that has been impacted over this week. At this time, they remain under a DDoS attack however they are taking the necessary steps to limit service disruptions. We continue to see significant improvements as they fine-tune their DDOS mitigation efforts, and we have confidence they will get their network restored to 100% capacity as quickly as possible.

We continue to collaborate with other ULCs to organize emergency ports from the impaired carrier to their services. It has become clear that the DDoS attacks are impacting the ability of Providers to port numbers. Providers are experiencing significant porting delays and we are also aware of the potential disruptions that could occur when moving numbers. We will continue to test and try alternative solutions to move beyond the control of the attackers.

Please understand this is not just an attack on a ULC but an attack on the United States Infrastructure and it has visibility at all levels of the federal government.
–END

Update October 1 4:15pm – All ULC services have been stabilized and calls are processing normally.  We performed a scheduled port today and it went through without issue. We believe this issue has now been resolved.

If you have any questions, please text 256-547-6817 or email support@internetpro.net.

09/01/21 8:09 PM: We have been informed of a widespread Comcast outage affecting a large number of our customers. Currently we are being told that they estimate services to be restored around 11:34 PM.

We will update this post if more information becomes available to us.

Update:  Services were restored pretty close to the expected time frame.  If you are still experiencing Comcast issues, please call or text us at 256-547-6817 or email support@internetpro.net.

Update 09/02/2021 2:00pm – The outage seems to be happening again.  It is affecting the same customers as well. Current ETR (estimated time to repair) is 5:51pm although we do see some devices coming bak online as of 2:12pm.

Update 3:25pm – It looks like most customers have come back online now.

Updated 9:30pm –  There was another outage from about 5:30pm to around 9:15pm affecting the same node.  We’ve been told that the issues are fiber related and have to do with the roll out of a new fiber product

July 28, 2020 – We’ve had several customers inquire about a phishing attempt which is apparently hitting a lot of our customers.  An email comes in which looks very similar to the screenshot below:

Please note that we do not have any such system and we never go by the name “Internetpro”.   We do own the domain internetpro.net, but the name of the company is Network Solutions, Inc.   If you have responded to this email and given your username and password, please contact us to change it immediately, as  your credentials have been stolen.

Update 8/14/2021:  Another variation is making the rounds.  It look similar to this:

Update 8/28/2021:  Yet another variation…Looks like this:

And Another (9/14/2021):

July 22, 2021 11:30am – There is a widespread internet outage going on.  We have seen issues reaching Microsoft as well as other services using the Akamai CDN (Content Delivery Network).  Akamai’s network touches many public services, and they may not be the primary source of the problem.   We will be posting updates here as we find out more.

This is not an issue with our servers or services and is something affecting the internet as a whole.

Update 11:50am: We are hearing that Amazon Web Services (AWS) is also affected.  Many large sites run on their services, so this is affecting banks, news sites, and others.

Update 11:55am: Down detector https://downdetector.com/ is showing 45 major sites on their homepage.  The site has to be reporting a major issue to be featured there. They specifically list a problem at Akamai as the root issue.

Update 12:05pm:  Akamai reports that they have implemented a fix and services are returning to normal.  We have seen sites coming back online as well.  We expect full normalization within the hour.

If you have any specific questions about this outage, please feel free to contact us via chat below, or text messaging or calling 256-547-6817 or by emailing support@internetpro.net.

We will be conducting planned maintenance on Saturday, July 24th.  This maintenance will mostly affect VPS (virtual private server) customers.  Email systems and shared hosting services will experience little to no interruption.   We will be upgrading the infrastructure for some servers and storage systems.   This maintenance will take longer than usual, so we will be beginning a bit earlier than normal, starting at 10:00pm CDT.

July 24, 2021 9:35pm – Maintenance has started.  Most work is being done “behind the scenes” right now and will not be customer-facing.

Update 10:15pm – Customer facing sites have begun to be affected.

Update 10:35pm – Email services *are* going to be limited for a brief time…Starting now.

Update 10:59pm – Email services restored.

Update 11:30pm – Because of the maintenance, we had taken our proactive alerting system offline (too many alerts would be sent).  However, we missed a shared server having trouble and affecting customer sites.  We’re working on that.

Update 11:40pm – The aforementioned unplanned outage has been resolved.

Update 12:35am – As part of the maintenance continues, our phone system is now offline.

Update 3:00am – Phone system back online and VPS machines are returning.

Update 3:35am – Everything seems to be functioning normally now.  If you have any services which are not performing correctly, please let us know.

If you have any questions or concerns about this work, please email us at support@internetpro.net, call or text 256-547-6817, or reach out through the chat box below.

July 13, 2021 – Sparklight is experiencing a major outage which started about 10:45 CDT.   It has been ongoing now for about 2 hours, so we assume this will be an extended outage.   The issue has taken down their call center as well as their website at https://www.sparklight.com.   Customers in the Gadsden area are also affected.  We have not been able to reach anyone for an ETR (Estimated Time to Repair) and currently are just following reports on Twitter.  We will post updates as we get them here.

Update 3:30pm – Their call center has had service restored.

Update 4:50pm – We are getting reports that Sparklight is beginning to come back online, but we definitely have customers still affected.  No additional information has been posted on their Facebook page.

Some articles related to the outage:
KTVB.COM
Valley News Live
Sparklight Facebook

• June 28, 2021 4:00pm – Comcast seems to be experiencing an issue in south Gadsden.  We began receiving calls as early as 2:30pm, so this seems be an outage that is going to be lasting a bit.  Currently, the estimated time to repair (ETR) is 7:14pm central time.  That time is usually automatically generated for 4 hours from the time they declare that an area is having an outage, so we doubt it is accurate.   Unfortunately, that is all of the information that we currently have.   This outage seems to primarily be affecting the Country Club area.   We’ll post more information here as we get it.

Update 6:25pm – The ETR is still listed as 7:14 p.m. No additional updates are available.

Update 6:35pm – We are told that modems are beginning to come online right now.

Update 7:00pm – Confirmed that things are back online.

May 29, 2021 2:55pm – One of our shared web servers is having some issues and sites it is hosting are currently unreachable.  We are investigating the problem to determine if it is a hardware failure, denial of service attack, or some other problem. Sites on that server will just not respond, so if you currently are having trouble accessing your web site, this is most likely the problem.   We will post updates here as we find out more information.

Update – 3:35pm – Services have been restored.  Unfortunately, we haven’t yet been able to tell exactly what the root cause of the issue was, but we are still investigating it.  If we find anything particularly interesting, we’ll post an update here.

If you have any further concerns regarding this outage or if your site has not yet come back online, please give us a call at 256-547-6817 or email support@internetpro.net.   It is a holiday weekend, but if you hit 8 for support, you will get a technician.