September 14, 2020 3:50pm – We are currently experiencing issues with our email system. Users are reporting that the server is not responding or is asking for passwords. We are investigating the problem and will post updates here.
Update 3:58pm – The issue has been tracked to our storage area network (SAN).
Update 4:05pm – We have implemented a fix. It will take a few minutes to resolve.
Update 4:31pm – Services appear to be restored. We’ll be monitoring closely for the next few minutes.
Update 4:34pm – For those wondering, no email was lost. It should be filtering into your inbox shortly. Services may be a bit slower for a few minutes as the system processes all of the requests from the last 45 minutes.
September 15, 11:40am – The issue has reoccurred and we are investigating. We believe we may have lost a network card on our SAN.
Update 12:22pm – The failure actually happened at 11:28am. We believe we have a hardware fault in the SAN. We’re currently doing a filesystem check to insure data integrity and then services will be brought back online. We are investigating the root cause.
Update 12:27pm – All services have been restored but we continue to investigate the cause so we don’t have a repeat of the issue.
Update 12:32pm – We’ve identified another issue and are pulling services offline again.
Update 12:42pm – Services are back online. Hopefully for real this time.
Update September 16th 5:30pm – The issue has happened yet again. Since it is slightly after business hours, we are going to do some major maintenance, not only on the email system but others as well. Web services as well as virtual private servers will be affected. We’ll be posting updates here.
Update 6:45pm – We were able to do some of the major maintenance without affecting virtual private servers and most web servers. We are continuing to work on the mail systems.
Update 7:20pm – We’ve implemented a fix as suggested by our SAN vendor. All services have returned to normal. We’ll have to wait to see if their fix actually takes care of the problem. Unfortunately, issues which occur every 24 hours or so are hard to troubleshoot. We do understand your frustration and definitely share it. We do appreciate your understanding.
Update – September 18th 9:35am – The issue has recurred. The good news is that we are getting good at recovering more quickly. We are in the process of getting hardware replacements.
Update – 1:35pm – Issue returned. We’re bringing it back up and expect recovery in 10-15 minutes.
Update – 1:45pm – Services have returned to normal. We have Saturday delivery equipment on the way. There will be maintenance tomorrow, but it should fix the issue.
Update – September 19th 5:00pm – Apparently our Saturday delivery did not get on the truck. We’re waiting to find out the status of our replacement equipment. Updates to follow as we get them.
Update – September 21st 12:10pm – The good news: Our equipment has been delivered to our technician’s home and he is in route with it. The bad news: The issue has recurred and we are int he process of recovery. Updates to follow.
Update 12:20pm – Services have been restored.
Update 7:55pm – We are starting replacement of hardware. Our hardware vendor maintains that there is a software problem, but the software configuration did not change in the 290 days of uptime the equipment had before failure, so we disagree. There is also the fact that their recommended reset did not work. We are replacing the hardware as our best troubleshooting step. There will be a bit of downtime until we can replace the existing hardware. We will be posting updates here as usual.
Update 8:25pm – All systems have been restored and we are operating on the new hardware. We’ll now see if the vendor is correct. Please bear with us as we troubleshoot these mission critical systems. We’ll soon be able to upgrade some systems with this new hardware. Stay tuned for that.
Update September 22, 2020 5:15am – Well, it looks like the vendor was correct in that they have a software error in their systems. We have reverted to the initial hardware after a failure of the system once again which occurred at approximately 4:15am. All systems have returned to normal operation and we can now concentrate on the software fix for the issue.
Update 11:25am – The system stopped responding again at 11:15am. We’ve restarted the troubled equipment and it should be online momentarily.
Update 11:35am – System came online at 11:31.
Update September 25th 5:30pm – The system failed again. We have been waiting for that and are implementing a recommended fix. We’ll post an update as soon as the system is back online.
Update 6:05pm – Services are back up. We took a bit longer on this one to make sure all file system checks were done and the implemented “fix” is complete. We’ll see if it actually is a fix.
FINAL UPDATE (hopefully) – October 3rd, 10:30am – The system has now been stable for over a week. We believe that it is truly fixed this time. We deeply regret any inconvenience this series of outages caused, and are glad to once again be able to count on the reliability of our email system. Thank you all for your patience as we worked through this difficult issue!
If you still have problems – We are seeing some people with Outlook having trouble getting reconnected. Our recommendation is to close Outlook for 2 minutes and then reopen. If it still doesn’t work, most likely a password has been input incorrectly. The newer versions of Outlook won’t ask for the password again, so you have to go into the server settings to put in the correct one. If you need help with that, please call us.
If you are still having issues related to this, please contact us via 256-547-6817 or by chat using the box to the lower right of your browser.