January 26, 2018 – We experienced some short connectivity issues early this morning. In general, the issues were under 5 minutes in duration and seem to have been caused by routing changes occurring upstream in response to failures outside our network. The connectivity issues varied in length depending on the networks involved (AT&T vs. Comcast for example).
One side effect of these issues also caused a problem with our email services. The anti-virus software tried to update during one of these intermittent connectivity issues and was not able to perform the upgrade properly. This caused the anti-virus software to crash. It looks like the issue was in place for about 1.5 hours. We did not detect the problem proactively because our monitoring systems test for server connectivity — not an actual successful sent email. The error that was produced was a “temporary error”, meaning that any email received during that time would be retried. As a result no email should be lost, but it may be delayed for a bit.
Update – We have received information that the issue caused by the anti-virus software was *NOT* caused by an internet disruption, but rather a bad virus definition update. This definitely fits better with our observation that the internet service disruption was *very* short lived and we saw no evidence of it starting as early as the virus software issue. It seems we were just a victim of two issues at the same time.
Update – 11:10am – The issue with the antivirus software happened again. We have restarted the software and it is back up and running. We are working with the software vendor to find a resolution.
Update – 11:18am – The bad update has been confirmed and our vendor has implemented a workaround until the signature database is updated. We don’t expect this issue to recur.
We apologize for any inconvenience this may have caused. If you have any questions about the issue, please email us at support@internetpro.net or call 256-547-6817.