News and Updates

November 12, 2019 –  Starting at approximately 1:15am CST, our systems detected an area wide outage of Comcast. This is affecting many of our customer phone systems, VPNs, and other services.  We are investigating.

Update 2:15am – Comcast seems to have cleared up the issue.  It lasted approximately 50 minutes, but systems are online now in the Gadsden area.  We’ve detected a few down systems outside of Gadsden but expect that they are due to the inclement weather.

October 22, 2019 – 8:00pm – AT&T is experiencing a *major* outage which is affecting many of our systems.  Phones are currently offline but are being redirected to tech cell phones as a failover.  Eventually, we’ll have time to actually implement real failover on our phone system so that doesn’t happen, but not today. 🙁    The outage is outside of our network, and we have not been able to get an estimated time to repair (ETR) from AT&T.   We are seeing reports online of the outages, and it seems to be widespread.  https://downdetector.com/status/att/

Update 9:45pm – After being on hold with AT&T, we’ve finally been issued a ticket to track the issue.  There is still no ETR, but it seems that AT&T is at least aware of the issue.

Update 10:00pm  – Service seems to be restored at least in this area.  We are seeing increased connectivity on our systems.

September 9, 2019 – We are experiencing some technical difficulties with our email server. The webmail interface is currently down.

Update: 12:37pm – Service is restored.

August 15, 2019 – We will be performing some upgrades on our email system which will briefly interrupt email connections.   We will be performing the maintenance in a window tonight between 6:00pm and 7:00pm.   We are doing this outside of our normal “late night” maintenance window so that software support services will be available in case of an error.

Most of the upgrades are going to be “under the hood” including an operating system upgrade.  There are going to be some changes to the web interfaces and to some system functionality, but most of the changes other than the web interface will go unnoticed by end users.  One small change that you may notice is that the calendar in Portal and Tuxedo will now fast forward or scroll to a time block relevant to ‘now’ and in addition, users will be able to set “work hours” which will also be displayed differently on week/day view.

We’ll be posting updates here as the work progresses.   If you have any questions, please contact us at support@internetpro.net, chat using the Online Support window in the lower right of your browser, or call us at 256-547-6817.

Update: 6:15pm – Maintenance has started and we’re waiting on a filesystem check to complete.   As soon as it does, services should be restored for the most part, with only brief interruptions expected.  The filesystem check has bene going for about 15 minutes now.

Update: 6:16pm – Services are back up.

Update 6:34pm – The upgrade ran into an issue.   Services are still online, but the upgrade has not completed on the server.   We are working to resolve the issues.

Update 6:43pm – Issues have been corrected.  Upgrade is continuing.  Services are still online.

Update 6:45pm – Services going offline for one final time.  We expect less than 5 minute interruption and the process will be complete.

Update 6:48pm – Services back online and the maintenance is complete.   We are doing “backend” checks, but there should be no more customer impact.

Update 6:57pm – It seems that there are some small cosmetic issues which we need to resolve, especially in the “Mail Settings” interface.  We’ll be working on those shortly, but the functionality is not impacted – just the look.

July 27, 2019 – We’ve had a couple of reports of email going to our users being delayed.  The sender may receive a message from the mail server with a message similar to the one below:

—BEGIN

The response was:

DNS Error: 49391460 DNS type ‘mx’ lookup of <domain hosted with us> responded with code NOERROR 49391460 DNS type ‘aaaa’ lookup of mail.internetpro.net. responded with code SERVFAIL 49391460 DNS type ‘a’ lookup of mail.internetpro.net. responded with code SERVFAIL

—END

We checked name lookups and initially found no problem.  However, after a few reports came in, we did find that our tertiary name server (ns3.internetpro.net) was not responding properly.   This seems to be only affecting IPv6 requests, and Google seems to prioritize those as they seem to be the only email provider having issues with our server.    So, users whose email is serviced by Google may receive the above message.  This is not a permanent error and the email will go through.

We are currently troubleshooting the issue affecting the name server and will post updates here.

Update 2:45pm – The server seems to be answering IPv6 queries correctly now.  We’re not entirely sure what caused the issue, but will be checking it periodically.   If you receive a delay like the one described above and it is time stamped *after 2:45pm today*, please let us know by forwarding it to support@internetpro.net.  We understand that you may receive one of these messages if you check your email after this time, but we are specifically looking for messages which reached your Inbox after this time.   Please make sure that the person receiving the delay message send it to support@internetpro.net with a different account.   They will most likely receive the same message sending to support@internetpro.net.   If needed, please call 256-547-6817 for help in getting one of these messages to us.

Update 9:00pm – We’ve had a report of this still happening.  As a preventative measure, we’ve taken the tertiary server and IPv6 offline.   This should force all queries to the other servers which do not seem to be giving this error.
Update 9:45pm – Our workaround has definitely worked as we are seeing delayed deliveries coming in now.  Even our support mailbox has received a few.
Update July 30 11:00am –  We are still getting reports of some affected domains.  This is mostly domains which are using our name services but not our email services.  We are addressing workarounds as we receive reports.   The update above regarding delivery to our servers still seems true as we haven’t had any further reports of that issue.
Update 2:25pm – We received a report from a customer with G-Suite services from Google.  They were able to chat with Google support.  Below is a partial transcript:

–BEGIN

Google Cloud Support, Iva: Upon checking it seems the MX records are not resolving for all servers
Google Cloud Support, Iva: That explains why you are able to get some emails but others are not delivered
Google Cloud Support, Iva: The MX records are there but they are not recognized by all servers
Google Cloud Support, Iva: It would be best to check with the domain host to see why exactly this is happening
<customer>: ok do I need to recheck the MX records for the domain?
Google Cloud Support, Iva: From what I see the records themselves are correct. The issue is with the fact some servers do not detect them

–END

The way we are reading this is that the support agent confirmed that our servers are giving correct answers but Google’s servers are not reading them properly.  Unfortunately, getting Google to fix the problem may be tough.

Update 7:15pm – We’ve gotten a Google reseller (Giorgio Torregrosa of PC-911 ) to open a ticket on our behalf!  We’re hoping for a call from someone at Google soon.  Thanks Giorgio!!
Update 8:15pm –  Google is able to reproduce the issue on an internal lookup tool and they see that it doesn’t happen on *any* tests outside of their system.  They have escalated the issue to find the problem on their end.

July 31 7:00am – We are still awaiting an update from Google
12:00pm – We reached out to Google via email for an update but have not heard back.
8:30pm – Google reports that their engineers were not able to reproduce or see a problem.  We have relayed that the issue is still ongoing and that they need to look into it some more.

August 1, 7:15am – Google has requested that we send them some “recent examples with timestamp and Msg-ID”.  This is pretty tough to do for us because almost everyone sends a screenshot from their phone which do not include these items.  If you know someone who has received a delay message, please have them forward the original *as an attachment* to support@internetpro.net.  Forwarding it as an attachment will allow the message to retain the items requested by Google.  Note, this message will need to come from the original sender.  Our customers can’t forward an example which may have been forwarded to them by the sender.

10:30am – We’re receiving reports from customers that email deliveries seem to be improved for them.  Since the delays sometimes last 24-48hrs, it is tough to say the issue is fixed, but we are getting some anecdotal evidence that things are better.

August 2, 7:00am – Google has reported that they are no longer able to reproduce the problem with their internal tools.  We have also received more reports that customers are successfully getting timely deliveries and have not received any reports of delays since late yesterday.   We *believe* the problem may be cleared up, but the nature of these delays is that they sometimes take 24-48 hours to be reported by Google, so right now we are crossing our fingers.   If anyone has a delay for a message sent starting in the afternoon of August 1 or later, please let us know by emailing support@internetpro.net.

August 3, 10:00am – We have not received a report of this happening for over 48 hours, so we do believe this issue has been resolved by Google.

June 18, 2019 – We will be scheduling some maintenance/downtime for Tuesday, June 25 from 11:30pm – 3:00am.   During this time, some or all services may be unavailable or intermittent as we perform hardware upgrades related to earlier, unplanned issues.   We will be posting updates as they are completed here.  Upgrades include new power systems, network card replacements, and upgrades to the network infrastructure and routing.   After the upgrades, network performance will be higher and more reliable.

Update June 25, 11:40pm – We are starting the maintenance.  Systems will be going offline soon.  We will post updates here as the maintenance progresses.

Update June 26, 1:56am – Maintenance continues to progress.  Shared hosting is online as well as POP/IMAP systems.  VPS systems and Hosted Exchange are currently offline.

Update 2:30am – All services should now be up and running normally.  We are doing final checks to make sure everything is working as expected.

Last update 3:30am – We did have one backup server which failed to spin up.  It will be out of commission until tomorrow, but only a few online backups are affected.  Otherwise, all services have been restored.

June 18, 2019 – Google calendar seems to be giving a “404 Not Found” error for everyone logged in.    This is an ongoing outage and news about what is going on is scarce currently.  We just wanted to alert our users that it is not a problem on your computer, phone, or other device – it is a Google issue.

One news story we have found can be seen here:  https://www.independent.co.uk/life-style/gadgets-and-tech/news/google-calendar-down-not-working-loading-error-404-outage-a8963851.html

Update – 10:25am –  https://mashable.com/article/google-calendar-is-down/

Update – 10:45am – They seem to be restoring services.  We can reach a calendar page now, but it is unable to load events.  Our understanding is that service restoration will vary by user/region.

Another article can be found here:  https://www.theverge.com/2019/6/18/18683625/google-calendar-down-worldwide-outage-404-error

Please use the articles above for further updates.  We will not be following this any longer as it doesn’t directly impact our services.

June 7, 2019 – Hot on the heels of the issues yesterday, we’ve experienced an issue with our hosted Exchange services.   Our engineers have isolated the problem, which is again a failed piece of hardware.   We are in the process of replacing that equipment.  We do believe that the hardware failure from yesterday damaged a few items including ports on servers and routers also connected to the failed switch.  We will be announcing a maintenance window to address some of these redundant hardware items in the near future.

Update – The issue was more complex than initially thought (sounds familiar).   We have restored service but are still working on the issue.  Service may be interrupted again.

Update 11:02am = Service has been fully restored.   If you are on hosted Exchange and are still having issues, please let us know by support chat, email support@internetpro.net or call 256-547-6817.

Update June 9 8:50pm – An issue with outgoing email was found and fixed.  Some emails sent since the earlier issue may have been delayed or failed.  If they were delayed, you will get a notice as such.  Failures will also be sent back to the sender.  This only affects email being sent FROM our servers.  Email coming TO our servers was not affected.

Thursday June 6, 2019 – We had network failure resulting from a core switch having an issue.   The outage lasted from approximately 1:11pm to 1:29pm while we tracked the issue and made the correction.  All services have returned to normal, and we do apologize for the interruption.   We do not expect any further problems from this issue.

Update:  As I posted the resolution, the problem reoccurred.  We are in the process of replacing equipment, but this does not seem to be a trivial issue.    Further updates to follow.

Update:  Definitely a core switch, but what we thought would be resolved with a reboot turned into a total hardware failure.  Because of the nature of the switch and the number of systems touched, this is affecting all services.  Some systems will take a bit of time to bring online.

Update: 2:24pm – Services are being restored now.  Most sites are up as well as email.  Some virtual servers are still awaiting recovery.

Update: Please note that our phone system and chat support are still being affected as they both reside on a virtual server which has not yet recovered.

Update: 2:42pm – Our phones and online chat are back.

Update 2:55pm – More and more VPS systems are coming online.   If you have specific status questions please use the support chat feature or email support@internetpro.net.

Update 3:15pm – It looks like all systems have recovered.   If you observe a problem with your website or service, please contact our technicians at 256-547-6817, by chat, or by emailing support@internetpro.net.