tnolen

August 3, 2022 – 6:15pm –  We experienced a service interruption as there were routing issues upstream of us.  This affected all services – phones, offsite backups, email, web hosting, etc.  The interruption started at about 5:00pm CDT and resolved itself at about 6:12pm.   We are continuing to monitor.  This issue is outside of our management area, and we are currently just able to work with upstream providers to troubleshoot.

Currently, this looks to be resolved as the routing has changed upstream – presumably to route around the troubled equipment or circuits.

Update 8:30pm – We had an additional disruption starting around 8:15pm.   This could be the routes resetting to the “normal” settings if the underlying issue has been fixed.  As of 8:30, everything seems OK, but we are continuing to monitor.

Update – August 4, 2022 – 6:45am – We have seen the issue return briefly a couple of times overnight.  Once around 1:15am and then a second time about 4:10am.  Each incident lasted approximately 15 minutes.  We still have an open ticket and are working to make sure the issue is properly fixed.

If you have any questions, please email us at support@internetpro.net.

July 20, 2022 – We have received some reports of users using Office 365 (Microsoft 365) services having trouble emailing our users.  The bounce back messages or non-delivery reports are stating that the issue is on the Microsoft side due to misconfigured PTR records.   A PTR record is similar to caller ID for computers. It allows one to look up the IP address and get a host name back.  Best practices state that this should be present for email delivery and in specific formats.  Here is an article about this for those of you who would like more information on it:  https://www.linuxmagic.com/best_practices/valid_ptr.html

It seems that Microsoft has made a change and is not following these practices at the moment.  We are looking into whether this is something that has been done by mistake and will be corrected or if they have done it on purpose for some reason.    We are currently very strict on PTR record validity because it does help us prevent spam.    Not all ISPs and email operators are as strict, so the Office 365 emails will go through to some addresses.    We will be posting updates on this as we have them.

Update 12:00pm – We have confirmed with our email vendor that this is a Microsoft issue that has been going on about 12 hours.  It is being discussed by the major email providers but Microsoft has not released a statement as yet, so there is not currently an estimated time to repair.

Update July 22 11:00am – We have not seen any statement from Microsoft on this, but have also not been seeing the issues continuing, so we do believe that Microsoft has corrected this on their end.  If you are still seeing the problem, please let us know by calling 256-547-6817 or emailing support@internetpro.net.

June 18, 2022 1:00pm – We are performing some required maintenance on the email system.  There will be some brief interruptions in client (email program) connectivity and webmail availability.  As part of this maintenance, we will also be performing some updates to security as well as usability including some modernizing of the webmail system.  No email will be lost, but there may be some slight delivery delays.  We’ll be posting updates here.  If you have any questions, please give us a call at 256-547-6817 as email to support@internetpro.net may be affected.

Update 1:50pm – This maintenance has been completed with very minimal interruption to customers.  If your email program had been left open during this maintenance, it is possible that it may prompt for a password.  Closing the email program and re-opening should resolve this.  As long as you do not enter a new password, the email program will remember the correct one which was being used.

Update 5:30pm – A customer notified us of an issue when connecting using SSL on port 465.   This was a result of our maintenance and has been corrected.

In initial testing, we are seeing a noticeable speedup in the web interfaces for our system.  In the coming months, we’ll be rolling out some additional features on the groundwork put in place today.   The older Tuxedo interface has now been replaced by the “New Tuxedo” version with more features and better performance.

If you have any questions or concerns about the work performed, please contact us at 256-547-6817 or email support@internetpro.net.

June 9, 2022 – We will be renewing a wildcard secure certificate today (*.internetpro.net).  This certificate protects/encrypts many pages and services including POP3, IMAP, SMTP, and web services on multiple servers.  There may be a brief interruption of these services as we load the new certificate.  This is a trusted certificate, so you should not get any errors during the process.  However, Apple devices sometimes do not handle certificate renewals well, so if you do get an error, please let us know by emailing support@internetpro.net, calling 256-547-6817, or using the chat window on this page.

We will post an update when the work is complete.

Update:  11:30am – This work has been completed.  We did not detect any issues or downtime at all.  Please let us know if you are experiencing issues which may be related.

April 16, 2022 – We’re doing some needed server maintenance this Easter weekend to minimize business impact.   During the maintenance, some websites may briefly go offline.   If you see anything unusual on your website, please let us know by emailing support@internetpro.net.   These updates do include some PHP version changes, but they should be non-breaking changes.

March 4, 2022 – 10:00am – One of our web servers has suffered a failure that is affecting some of our shared hosting sites.  We are working to address the problem.  Originally, a RAID array lost a drive and it was replaced as normal, but apparently the rebuild process failed and we are seeing degradation in service.  We will be posting updates here as we have them.

Update 1:45pm – The server with the failed RAID array is currently offline as we work to repair the issue. Some sites will be offline until this process is completed. We will update this post with further information as we continue the repairs.

Update 6:45pm – We’ve recovered the filesystem, but unfortunately have some corruption in the database system of the affected server.  We’re working on restoring from backups but are running into some corruption issues.

Update 10:30pm  – We have successfully restored a few sites and have a procedure for restoring the rest.  Sites will be coming up as we restore them.

Update 11:15pm – We believe all sites except for one have been restored to full functionality and we have informed the owners of the affected site.   If you are still experiencing an issue with your website, please email us at support@internetpro.net to let us know and we will investigate it further.  We do apologize for this issue.

March 5, 2022 – 8:30am – We’re going to attempt one last effort to get the one website still having issues recovered.  The effort will affect all other sites briefly.  We’ll update here as to the success or failure and the conclusion of the event.

Update 8:50am – Failure.   The site in question was in the midst of a redesign, so we’re pressing forward with that.  There should be no more service interruptions related to this outage.

February 15, 2022 – We’re seeing a new batch of phishing emails seeking to harvest usernames and passwords from the email system. Please remember that we will never send out emails asking you to verify your passwords.  It is important to stay diligent regarding these types of communications – not just with our email but with things like your bank account, taxes, social media, or any other login you have.  Here is an example of what is being sent today:

If you have any questions or have already interacted with this email, please email us at support@internetpro.net, call 256-547-6817, or use the chat window in the lower right of your screen.

January 17, 2022 – Well, we made it through 2 years of the pandemic before Covid hit our office, but it has come with a vengeance.  For the next few days, we may not be able to respond as quickly as normal for service calls due to a large portion of our staff being sick and/or quarantined.   We will be doing as much as we can using our remote tools such as iFoye to reduce further exposure.   Please bear with us as we work through the process of getting our staff better.

Some staff working remotely are also having a bit of trouble talking on the phone due to coughing, so if possible, we recommend the chat box at the lower right of your browser or email support@internetpro.net if possible.

Normally, the virus icon is reserved for computer viruses, but I thought it was appropriate today…

Thank you for your understanding and patience.

Update January 18, 2022 – We have yet another staff member out today, so our ability to respond continues to be limited.

Update January 21, 2022 – More staff members are out.  We will most likely be closed today for lunch from 1:00 to 2:00pm.  We apologize for the inconvenience.

January 1, 2022  12:00pm – Happy New Year!  The new year brings more of the same old problems.  In this case, a bug in the anti-malware engine of Microsoft Exchange servers has caused email delivery to stop.  Customers who have their own on premise Exchange server will be affected by this.   There is no official patch just yet.   The bug is in the anti-malware service on these servers and the current work around is to stop this service.  You can find an article with more information about this bug here.

Email delivery will be delayed and senders who send to people using these Exchange servers may/will receive a “delayed delivery notification”.   As long as the issue has been addressed within a few days, those emails will eventually be delivered.  If any sender has a short retry time setup on their email, they will receive a second “failure notice” indicating that the email did not go through and will not be tried any longer.   The industry standard is for email delivery to be retried 5 days, but this can be adjusted by individual email operators.

If you have any questions or concerns, please email us at support@internetpro.net. Please keep in mind that our office is closed for the holiday, so you may not receive an immediate response.   We are working on implementing the workaround for customers with proactive support contracts and those who report the issue to us and wish it fixed at holiday/after hours rates. This will allow the email to be delivered.  For others, we will be rolling out the workaround or fix if it has been rolled out during normal business hours Monday.

Update – Microsoft has released an emergency fix which does require manual application.   We are working to put this fix in place at normal rates for affected customers.