News and Updates

January 17, 2022 – Well, we made it through 2 years of the pandemic before Covid hit our office, but it has come with a vengeance.  For the next few days, we may not be able to respond as quickly as normal for service calls due to a large portion of our staff being sick and/or quarantined.   We will be doing as much as we can using our remote tools such as iFoye to reduce further exposure.   Please bear with us as we work through the process of getting our staff better.

Some staff working remotely are also having a bit of trouble talking on the phone due to coughing, so if possible, we recommend the chat box at the lower right of your browser or email support@internetpro.net if possible.

Normally, the virus icon is reserved for computer viruses, but I thought it was appropriate today…

Thank you for your understanding and patience.

Update January 18, 2022 – We have yet another staff member out today, so our ability to respond continues to be limited.

January 1, 2022  12:00pm – Happy New Year!  The new year brings more of the same old problems.  In this case, a bug in the anti-malware engine of Microsoft Exchange servers has caused email delivery to stop.  Customers who have their own on premise Exchange server will be affected by this.   There is no official patch just yet.   The bug is in the anti-malware service on these servers and the current work around is to stop this service.  You can find an article with more information about this bug here.

Email delivery will be delayed and senders who send to people using these Exchange servers may/will receive a “delayed delivery notification”.   As long as the issue has been addressed within a few days, those emails will eventually be delivered.  If any sender has a short retry time setup on their email, they will receive a second “failure notice” indicating that the email did not go through and will not be tried any longer.   The industry standard is for email delivery to be retried 5 days, but this can be adjusted by individual email operators.

If you have any questions or concerns, please email us at support@internetpro.net. Please keep in mind that our office is closed for the holiday, so you may not receive an immediate response.   We are working on implementing the workaround for customers with proactive support contracts and those who report the issue to us and wish it fixed at holiday/after hours rates. This will allow the email to be delivered.  For others, we will be rolling out the workaround or fix if it has been rolled out during normal business hours Monday.

Update – Microsoft has released an emergency fix which does require manual application.   We are working to put this fix in place at normal rates for affected customers.

December 8, 2021 – The 3CX company has updated the Chrome extension many of our customers were using.  They actually have replaced the old extension with a new “Click2Call” extension.  The new extension does not have the same functionality as the old one with the small window showing phone controls, presence, etc.   The new extension only allows you to click phone numbers in your browser to initiate a call.   The full web client is still available, but for those who want the smaller popup window, you will need to install the new Windows desktop app.

For customers under maintenance with Network Solutions, Inc., we’ll be glad to provide free assistance and instruction for updating to the new desktop app for Windows users.  Users of Mac or Linux will need to use the web client as the desktop app is not available.  It is possible to load the old extension for Mac and Linux, but it will stop working completely in June 2022 when it is removed from the Chrome app store.

To install the desktop app yourself, simply navigate to the web client (most of our customers have this bookmarked).  It is something along the lines of https://companyname.al.3cx.us:5001/webclient usually.    Once logged in, there will be a Microsoft logo in the upper left.  Click that to pop up an instruction windows.  There are two steps: 1) Install and 2) Provision (activate).

Installing will download a program to run. Once that is run and installed, click the second step and the app will provision itself.

If you have any questions or need assistance, please contact our office at 256-547-6817 or use the Chat window below.   Please note that this was not a  scheduled update, so we have many unexpected requests for help.  We may need to schedule a return call as a result.

December 7, 2021 –  AWS is the underlying infrastructure for a good portion of the web.  As such, the degradation of their services is affecting MANY sites and services.  These include, but are not limited to Control4’s 4Sight platform, social media websites, cloud software providers including practice management and accounting programs, and much more.  Directly affected are Amazon’s own services including Ring, Alexa, and their shopping/shipping services.  If you are having an issue online, chances are it is related to this ongoing issue.

Here are some links which provide more information, including details about today’s outage:  https://apnews.com/article/amazon-web-services-outage-978394126305

The AWS services dashboard:  https://status.aws.amazon.com/

Downdetector is a good source showing many sites which are reporting related issues:  https://downdetector.com/

Here at Network Solutions, Inc., we don’t rely on any AWS platforms, so our services are unaffected.

September 27, 2021 – 9:30am – We are experiencing an issue with a shared web server which is affecting some customers.   We are currently investigating the issue as it is taking longer to recover than we initially thought.  We’ll be posting updates here as we get them.

Update 10:03am – Services have been restored.  A combination of DOS (denial of service) attack and hardware failure was responsible.  We are working on a hardware fix.

If you have any questions or concerns about this, please email us at support@internetpro.net, text or call 2565476817, or use the chat box in the lower right of your screen.

September 25, 2021  3:30pm  – We are getting reports of our customers not being able to receive calls from Verizon.  Right now, only Verizon has been tested.   We’re working on other tests to see if we can narrow down the issue.

We will post updates here as we have them.

Update 3:40pm – We have confirmed that this is a Verizon issue.   It is affecting Verizon wireless customers and customers of cellular services which use their networks such as Spectrum and Straight Talk.   Downdetector is also reporting Verizon issues here: https://downdetector.com/status/verizon/.

Update  3:55pm – It looks like the worst has passed.   Downdetector is showing that the reports of issues are starting to drop and we have had reports of Verizon calls going through to our customers now.

September 27, 2021 – 10:00am – We are not sure the issues are related to this weekend’s events, but our carrier has posted a bulletin stating this:
“NOTICE: Currently a large underlying carrier is experiencing intermittent service disruption. This outage is causing inbound calls of some numbers to not be delivered to SIPTRUNK. We apologize for any service interruption that these issues may cause. “

Update 11:30am – We have discovered that DTMF tones are no longer working on digital receptionists.  DTMF tones are the tone made by your phone when you dial on the keypad and they are used to choose options on the menus in phone systems (for instance, push 1 for sales, push 2 for support, etc.).   Because of this, people are not able to make choices on those menus.  Some customers have their systems set to hang up if no choice is made to weed out telemarketers.  Right now, those systems may be weeding out everyone.  If this seems to be the case on your system, please contact us and we can change the system so that “no choice made” goes somewhere. We have alerted our carrier to the issue in case they are unaware.

Update 11:48am – This seems to be a widespread issue as we have now received a report from a customer using AT&T as the carrier who is having similar issues.

Update 12:00pm from SIPTRUNK – “Our carrier is experiencing an intermittent issue with inbound and outbound calls. This includes audio & DTMF tones. But static is usually only introduced at analog equipment, so we suggest rebooting any physical devices you might have, including router, modems, firewalls, PBXs, etc. But we have multiple tickets open with the carrier. All teams are actively engaged at this time. Currently no ETA. We will provide an update as soon as one is available.”

Update 3:45pm – SIPTRUNK reports that the issue is still under investigation.

Update 6:35pm – Our carrier reports that the network behavior has returned to normal.  We are changing phone system programming that was changed because of the DTMF issues.   Please let us know if you are still experiencing abnormal phone system behavior.

Update September 28 8:45am – Although the carrier reported that things were back to normal, we are still receiving reports or issues (and experiencing them ourselves).   We have reported this and are awaiting news.

Update 4:30pm – Updates have been slow to come today.   Our carrier has given us a bit more information now though.  They are seeing about 1/7 the tickets today as yesterday, so things are improving.   The issue is not a misconfiguration, but rather is a DDOS attack directed at VOIP carriers.  The affected carriers have not released that it is a ransomware attack, but that happened recently to a large Canadian carrier last week and the rumor is that this is related and may be the same attackers.   Earlier, Verizon was the suspected source of the issue and even Verizon themselves thought they were the problem.  However, the underlying carrier currently being attacked is Bandwidth. Because of the interconnections between companies, it seemed to be a Verizon problem at first because they had the most end users affected.   Bandwidth is a large “behind the scenes” carrier.   You can check their Down Detector information here: https://downdetector.com/status/bandwidth/   and their own status page here: https://status.bandwidth.com/.   We expect them to post “everything normal” later today as off-hours voice traffic drops a lot and reduces the impact of the attacks, but we may see the same issues return tomorrow if they have not been able to mitigate the attacks.

Update September 29  8:45am – Currently, it looks like a repeat of yesterday, where daytime business traffic is starting to have an impact on performance of the network.  Downdetector is showing an increase of reported issues again: https://downdetector.com/status/bandwidth/.   Our carrier is working on diversifying their routes to other upstream carriers, but this involves a lot of manual porting of hundreds of thousands of numbers, so this will take some time.   Mitigation of the attack at the carrier level is improving, so we expect less problems as time goes on, but there *are* still issues happening.

Update 10:45am – Carrier has posted an issue this morning but are currently saying it only affects inbound calls from the Verizon network.  Of course, other providers using the Verizon network such as Spectrum Mobile, Straight Talk, etc. will also be affected.  Here is another recent news article with a little more information:  https://www.zdnet.com/article/bandwidth-ceo-confirms-outages-caused-by-ddos-attack/

Update 4:00pm – Siptrunk sent out a letter regarding the issue.  No real extra news, but some clarifications.   Here is an excerpt from the letter we received:

–BEGIN

As you may be aware, one of the US’s largest underlying carriers (ULC) has been under a Distributed Denial-of-Service (DDOS) attack this week. This attack is similar but larger in scope than the attack on VoIP.MS that started on September 16th.    Since this is a ULC we can easily route around them on outbound calls, but this is not the case for inbound calls. The ULCs serve as the “carrier of record” for the DIDs or telephone numbers used by our customers. The current design of the US telephone network means that all calls destined to a particular telephone number have to be handled by the carrier of record. The only way to change this is to port or move the number to a new carrier of record and for landlines, this process can take 5 to 7 days under normal circumstances.

SIPTRUNK is investigating all options to address intermittent inbound call issues caused by the DDOS on the ULC, but please know this is affecting large and small customers the likes of which including Microsoft Teams, Google voice, RingCentral, etc… to just name a few.

We have seen significant improvement on the ULC that is being attacked, and we have confidence they will get their network restored to 100% capacity as quickly as possible.

However, with that said SIPTRUNK has been working with other ULCs to arrange porting of DIDs to alternative ULCs. As you can imagine there are a number of carriers and customers attempting mass ports and this is causing congestion in the porting process. We will continue to work to have inbound calls delivered via alternative ULCs.

–END

September 30 – 7:15am – For the technically curious, here is a good video between a well-known MSP (Managed Service Provider) and a VOIP provider on Youtube discussing the issue.  It uses a lot of jargon, but may be interesting to some of you.  https://www.youtube.com/watch?v=CeCd9zW-R5o.  It is from yesterday, so when they say “yesterday”, they are talking about the 28th.  However, not much has changed.

Update 1:45pm – Upstream providers report much better stability of the network today.  There are still some users reporting issues, but the number and severity seems to be diminished.

Update 4:15pm – SIPTRUNK has released another statement.  Here are the relevant parts:
–BEGIN
All SIPTRUNK Network Services remain operational. We continue to monitor the underlying carrier (ULC) that has been impacted over this week. At this time, they remain under a DDoS attack however they are taking the necessary steps to limit service disruptions. We continue to see significant improvements as they fine-tune their DDOS mitigation efforts, and we have confidence they will get their network restored to 100% capacity as quickly as possible.

We continue to collaborate with other ULCs to organize emergency ports from the impaired carrier to their services. It has become clear that the DDoS attacks are impacting the ability of Providers to port numbers. Providers are experiencing significant porting delays and we are also aware of the potential disruptions that could occur when moving numbers. We will continue to test and try alternative solutions to move beyond the control of the attackers.

Please understand this is not just an attack on a ULC but an attack on the United States Infrastructure and it has visibility at all levels of the federal government.
–END

Update October 1 4:15pm – All ULC services have been stabilized and calls are processing normally.  We performed a scheduled port today and it went through without issue. We believe this issue has now been resolved.

If you have any questions, please text 256-547-6817 or email support@internetpro.net.

09/01/21 8:09 PM: We have been informed of a widespread Comcast outage affecting a large number of our customers. Currently we are being told that they estimate services to be restored around 11:34 PM.

We will update this post if more information becomes available to us.

Update:  Services were restored pretty close to the expected time frame.  If you are still experiencing Comcast issues, please call or text us at 256-547-6817 or email support@internetpro.net.

Update 09/02/2021 2:00pm – The outage seems to be happening again.  It is affecting the same customers as well. Current ETR (estimated time to repair) is 5:51pm although we do see some devices coming bak online as of 2:12pm.

Update 3:25pm – It looks like most customers have come back online now.

Updated 9:30pm –  There was another outage from about 5:30pm to around 9:15pm affecting the same node.  We’ve been told that the issues are fiber related and have to do with the roll out of a new fiber product

July 28, 2020 – We’ve had several customers inquire about a phishing attempt which is apparently hitting a lot of our customers.  An email comes in which looks very similar to the screenshot below:

Please note that we do not have any such system and we never go by the name “Internetpro”.   We do own the domain internetpro.net, but the name of the company is Network Solutions, Inc.   If you have responded to this email and given your username and password, please contact us to change it immediately, as  your credentials have been stolen.

Update 8/14/2021:  Another variation is making the rounds.  It look similar to this:

Update 8/28/2021:  Yet another variation…Looks like this:

And Another (9/14/2021):

July 22, 2021 11:30am – There is a widespread internet outage going on.  We have seen issues reaching Microsoft as well as other services using the Akamai CDN (Content Delivery Network).  Akamai’s network touches many public services, and they may not be the primary source of the problem.   We will be posting updates here as we find out more.

This is not an issue with our servers or services and is something affecting the internet as a whole.

Update 11:50am: We are hearing that Amazon Web Services (AWS) is also affected.  Many large sites run on their services, so this is affecting banks, news sites, and others.

Update 11:55am: Down detector https://downdetector.com/ is showing 45 major sites on their homepage.  The site has to be reporting a major issue to be featured there. They specifically list a problem at Akamai as the root issue.

Update 12:05pm:  Akamai reports that they have implemented a fix and services are returning to normal.  We have seen sites coming back online as well.  We expect full normalization within the hour.

If you have any specific questions about this outage, please feel free to contact us via chat below, or text messaging or calling 256-547-6817 or by emailing support@internetpro.net.