News and Updates

Sept 3, 2010 – We are experiencing an outage that started at 4:00pm.  We are working with AT&T on a resolution.  The ticket is currently in the testing stages and we do not know what the upstream issue is.   The problem is definitely outside our network, and we are waiting on news from the upstream providers in order to get an estimated time to repair.

Update 7:50 pm.  Looks like everything is back up. AT&T has still not informed us of what the problem was, but because of the Labor Day weekend they were slower than usual in their response.  We are working on adding an additional provider to add more redundancy since we cannot count on them.

Sept 4, 2010 – AT&T never did contact us about the resolution to the problem and they automatically closed their ticket.  However, this morning we are continuing to experience issues related to this problem.   As of 8:50am, most sites/services are being impacted.  We have re-opened a ticket with AT&T and are hoping they will work on this in a timely manner, but right now it does not seem like they want to work on a holiday weekend.

Update 11:25am  – Finally got some needed info from AT&T.  They have a major fiber cut between Anniston and Talladega due to construction work.   Estimated Time to Repair is 1:00pm CDT.   This issue is affecting a large area.

Update 11:31am – Well, that was quick.  They just told us 1:00, but everything just came online.

We have upgraded our alternate webmail interface.   Our alternate interface can be found at https://webmail.internetpro.net.   We have just upgraded it to the newest version available.  This interface includes several advantages over our traditional interface, including an HTML editor and drag-and-drop ability using AJAX.   However, the alternate interface is not integrated into your email settings or your server-side filtering.  Feel free to try out either interface to see which one suits your needs better.

Update: We have added a button on our home page under the Online Email Center which links to the alternate interface. We hope you enjoy it.

As of 9:50am, our phones have gone out and we are unable to make outgoing calls or receive incoming calls. We are working with our telco provider to resolve the issue. If you have an immediate need, please email us at support@internetpro.net with a callback number.

10:20am – Our phone system is back online and we do not anticipate any additional issues.

We are replacing some hardware that is about to fail in one of our web servers.  The server will go down for a short time and will affect some web hosting customers.  The downtime should be very short, and this is a necessary replacement to keep the server from failing later.  – 10:20pm

Server back online at 10:28.  We have tested it and all sites look OK.  Eight minutes downtime.

At 4:22pm we experienced an outage of our primary circuit. It is an upstream outage and seems to be causing issues with Verizon phones and multiple carriers as well.  We are working with upstream providers on a resolution.

5:19pm – The issue is a major fiber cut between Birmingham and Talladega.  There is no current estimated time to repair.

10:15pm – Estimated Time to Repair is now 3:00am.  AT&T has identified two problems — the fiber cut and a hardware issue caused by the bouncing circuit.  If they fix either one or the two, our circuits should normalize.

3:25am – Service seems to be fully restored.  Some networks/carriers still affected by the issue, and complete restoration of all fiber is now estimated to be repaired by 4:00am CDT.

Don’t forget:  You can follow updates such as these on Twitter by following @netsolinc.

One of our wireless base units is experiencing a problem.  This is only affecting a few customers.  The first issue was reported at 4:27pm and was “fixed” at 4:43pm, but the unit is continuing to fail.  We are in the process of diagnosing the problem and rerouting affected customers to our redundant wireless base units. It is currently 6:15pm.

All customers are have been successfully rerouted to redundant systems.  The primary system is currently operational as well, but we will be testing it to find the issue causing the sporadic malfunction. 6:30pm

We are experiencing a problem with one of our web servers.  The issue was caused by a failed security update. Sites affected by this issue may get an error saying that a required PHP component is not installed.  We are working to resolve this issue. 7:15am

Update 7:31am:  Some sites are back online.  We are still experiencing issues with the server, but on most sites those issues should be unnoticeable.

Update 7:55am:  We have gone through all the sites on the server, and they all seem to be functioning properly.  We are still getting some errors on the server, which we will be working to correct, but in the mean time, all sites are online.

One of our spam servers had an issue overnight. It did not thrown any alarms because the service was “partially” running.  We noticed more spam than usual making it through the system this morning, so we did some checking.  After reviewing the message headers, we determined that the spam getting through was not actually being checked by the system.  Upon further investigation, we found a server malfunctioning.  The issue has been resolved now, so your spam levels should return to normal.

We are presently experiencing issues with one of our virtual server hosts.  We are working on resolving the problem.  Some customers with VPS’s are affected.
Some websites (hosted on virtual servers) are also affected.

We will post updates here. 2:56pm

Update: The problem is stemming from a failure of the shared NAS which hosts the virtual server storage.  Efforts are underway to restore NAS service. 3:39pm

Update: Most VPS are back online.  We are checking all servers individually to make sure they are operating properly. 3:47pm

Update: Looks like all VPS machines are up and accounted for.  We are working on finding cause.  The NAS networking shut down, and that caused the VPS’s to not be able to write to their filesystems.  We are not yet sure why the networking failed on the NAS. 3:54pm

AT&T Network Engineers will be performing a planned maintenance on
Birmingham-Alabama Gigabit Access Router 1. The work will be performed on 7/1/2010,
during the maintenance window of 12:00AM-6:00AM, local time.
AT&T's goal is to successfully complete the task and restore service within the
stated time frame. 

We apologize for any inconvenience that you may experience.
We expect service to be impacted somewhat during the maintenance window.

AT&T Network Engineers will be performing an IOS upgrade.

Please Note: This maintenance is IN ADDITION TO the planned maintenance on 6/18/2010.
The work on 6/18/2010 will be done on Birmingham-Alabama
Gigabit Access Router 3, while this is being done on Router 1. The notice for that
maintenance can be found at https://www.netsolinc.com/planned-maintenance.

Update: The 7/1/2010 maintenance has been rescheduled to 7/6/2010.  The
maintenance window is the same as is the scope of work.

Update: The 7/1/2010 maintenance which was moved to 7/6/2010 has been moved again.
The new maintenance date is 7/9/2010.  The time window and the scope of work
remains the same.

Update: The maintenance was completed.  Service was affected in the expected manner
and we had some degradation of performance from 2:25AM to 3:05AM CDT.