News and Updates

One of our wireless base units is experiencing a problem.  This is only affecting a few customers.  The first issue was reported at 4:27pm and was “fixed” at 4:43pm, but the unit is continuing to fail.  We are in the process of diagnosing the problem and rerouting affected customers to our redundant wireless base units. It is currently 6:15pm.

All customers are have been successfully rerouted to redundant systems.  The primary system is currently operational as well, but we will be testing it to find the issue causing the sporadic malfunction. 6:30pm

We are experiencing a problem with one of our web servers.  The issue was caused by a failed security update. Sites affected by this issue may get an error saying that a required PHP component is not installed.  We are working to resolve this issue. 7:15am

Update 7:31am:  Some sites are back online.  We are still experiencing issues with the server, but on most sites those issues should be unnoticeable.

Update 7:55am:  We have gone through all the sites on the server, and they all seem to be functioning properly.  We are still getting some errors on the server, which we will be working to correct, but in the mean time, all sites are online.

One of our spam servers had an issue overnight. It did not thrown any alarms because the service was “partially” running.  We noticed more spam than usual making it through the system this morning, so we did some checking.  After reviewing the message headers, we determined that the spam getting through was not actually being checked by the system.  Upon further investigation, we found a server malfunctioning.  The issue has been resolved now, so your spam levels should return to normal.

We are presently experiencing issues with one of our virtual server hosts.  We are working on resolving the problem.  Some customers with VPS’s are affected.
Some websites (hosted on virtual servers) are also affected.

We will post updates here. 2:56pm

Update: The problem is stemming from a failure of the shared NAS which hosts the virtual server storage.  Efforts are underway to restore NAS service. 3:39pm

Update: Most VPS are back online.  We are checking all servers individually to make sure they are operating properly. 3:47pm

Update: Looks like all VPS machines are up and accounted for.  We are working on finding cause.  The NAS networking shut down, and that caused the VPS’s to not be able to write to their filesystems.  We are not yet sure why the networking failed on the NAS. 3:54pm

AT&T Network Engineers will be performing a planned maintenance on
Birmingham-Alabama Gigabit Access Router 1. The work will be performed on 7/1/2010,
during the maintenance window of 12:00AM-6:00AM, local time.
AT&T's goal is to successfully complete the task and restore service within the
stated time frame. 

We apologize for any inconvenience that you may experience.
We expect service to be impacted somewhat during the maintenance window.

AT&T Network Engineers will be performing an IOS upgrade.

Please Note: This maintenance is IN ADDITION TO the planned maintenance on 6/18/2010.
The work on 6/18/2010 will be done on Birmingham-Alabama
Gigabit Access Router 3, while this is being done on Router 1. The notice for that
maintenance can be found at https://www.netsolinc.com/planned-maintenance.

Update: The 7/1/2010 maintenance has been rescheduled to 7/6/2010.  The
maintenance window is the same as is the scope of work.

Update: The 7/1/2010 maintenance which was moved to 7/6/2010 has been moved again.
The new maintenance date is 7/9/2010.  The time window and the scope of work
remains the same.

Update: The maintenance was completed.  Service was affected in the expected manner
and we had some degradation of performance from 2:25AM to 3:05AM CDT.

AT&T tells us that they are having an outage of some kind.  Customers have line sync, but are not able to authenticate.

AT&T expects service to be restored by midnight tonight.

AT&T Network Engineers will be performing a planned maintenance on
Birmingham-Alabama Gigabit Access Router 3. The work will be performed on 6/18/2010,
during the maintenance window of 12:00AM-6:00AM, local time.
AT&T's goal is to successfully complete the task and restore service within the
stated time frame. 

We apologize for any inconvenience that you may experience.
We expect service to be impacted somewhat during the maintenance window.

AT&T Network Engineers will be performing an IOS upgrade.

We launched a new version of our web site!  Much of the information and functionality is the same, but we are updating the “look”.   Also, the new site is built on WordPress technology, which allows easy and quick updates of the site.   We are also tying the site in with Twitter.  You can follow us @netsolinc.   We will post updates of new products, specials, or events that may affect your service.  We hope you enjoy the new look and your feedback is welcomed.  If you have any issues with the site, please let us know!

We have one host down in our offsite datacenter.  It looks like a connectivity problem with the datacenter rather than a host issue.  We are working to resolve the issue.

Update 6:48pm – We have confirmed that it is yet another problem with a Cisco switch.  Datacenter engineers are working with Cisco now to resolve the issue.  Network connectivity has been intermittent since 6:37pm.

Update 8:40pm – Switching has been relatively stable for the last hour or so, but the datacenter engineers are continuing to research and diagnose the issue with Cisco TAC.

Update – Cisco identified a software bug.  They have applied updates to the equipment and the network is stable now.

Post Incident Report
Date / Time of Incident: 6/1/2010 6:44PM ET
Duration of Incident: 1hour 13minutes
Scope of Incident: Newark, Delaware Data Center Dedicated, Colocated and Cloud Customers

Description of Incident:
At 6:44PM ET one of the two redundant Cisco 6509 switches supporting the Dedicated, Colocated and Cloud Customers failed.  The second switch failed in the same manner when load from the first switch was re-routed to the second.  The end result was loss of connectivity for approximately 40 minutes.  As the switches recovered, latency lasted another 33 minutes.

Actions During Incident:
6:44 PM  - Network Operations and Network Engineering were immediately dispatched to investigate and resolve the issue.  A ticket was opened with the vendor of the switches (Cisco) and efforts to restore service were coordinated between the datacenter and Cisco.  

6:55 PM - After exhausting normal troubleshooting avenues, the Switch 1 is reloaded at console by Network Engineers

6:58 PM – Switch 2 primary supervisor engine fails to boot, secondary supervisor engine becomes active

7:03 PM – Switch 1 completes rebooting , routing protocols established, datacenter team monitors performance of switch

7:08 PM – Switch 2 Primary supervisor engine is manually reset, secondary supervisor engine becomes fully active

7:12 PM – Switch 1 primary supervisor engine fails, secondary supervisor begins to take over

7:20 PM – Switch 1 secondary supervisor takes over

7:37 PM – Switch 1 Primary supervisor engine pulled, engineers suspect faulty hardware

7:48 PM – Switch 2 Secondary Supervisor engine completes booting, begins forwarding traffic.  CPU on both switches remain elevated for 10 minutes while routing protocols are re-established

7:58 PM – Forwarding returns to normal, data delivered to Cisco for further analysis

Root Cause of Incident:
Cisco has identified a specific bug as the likely root cause that created a memory leak which triggered the event.  Still awaiting final confirmation from Cisco on the bug, but initial findings indicate software issues as the root.

Further Actions:
Based on an expected confirmation from Cisco on the bug, a software patch will be applied in a lab environment.  After successful application of the patch, Network Engineers will be applying the patch during a scheduled and announced maintenance window.

The datacenter Network Engineering team is looking into other architectural changes to add additional diversity to the customers that are currently supported by the current set of Cisco 6509s to further insolate customers from any future type of single core switching failure.

Our phone lines have all been ported over to voice over IP.   Preliminary testing looks good.  If you are interested in this type of system at your own office, please contact us.  It can save you a lot of money on your phone lines.