News and Updates

We are presently experiencing issues with one of our virtual server hosts.  We are working on resolving the problem.  Some customers with VPS’s are affected.
Some websites (hosted on virtual servers) are also affected.

We will post updates here. 2:56pm

Update: The problem is stemming from a failure of the shared NAS which hosts the virtual server storage.  Efforts are underway to restore NAS service. 3:39pm

Update: Most VPS are back online.  We are checking all servers individually to make sure they are operating properly. 3:47pm

Update: Looks like all VPS machines are up and accounted for.  We are working on finding cause.  The NAS networking shut down, and that caused the VPS’s to not be able to write to their filesystems.  We are not yet sure why the networking failed on the NAS. 3:54pm

AT&T Network Engineers will be performing a planned maintenance on
Birmingham-Alabama Gigabit Access Router 1. The work will be performed on 7/1/2010,
during the maintenance window of 12:00AM-6:00AM, local time.
AT&T's goal is to successfully complete the task and restore service within the
stated time frame. 

We apologize for any inconvenience that you may experience.
We expect service to be impacted somewhat during the maintenance window.

AT&T Network Engineers will be performing an IOS upgrade.

Please Note: This maintenance is IN ADDITION TO the planned maintenance on 6/18/2010.
The work on 6/18/2010 will be done on Birmingham-Alabama
Gigabit Access Router 3, while this is being done on Router 1. The notice for that
maintenance can be found at https://www.netsolinc.com/planned-maintenance.

Update: The 7/1/2010 maintenance has been rescheduled to 7/6/2010.  The
maintenance window is the same as is the scope of work.

Update: The 7/1/2010 maintenance which was moved to 7/6/2010 has been moved again.
The new maintenance date is 7/9/2010.  The time window and the scope of work
remains the same.

Update: The maintenance was completed.  Service was affected in the expected manner
and we had some degradation of performance from 2:25AM to 3:05AM CDT.

AT&T tells us that they are having an outage of some kind.  Customers have line sync, but are not able to authenticate.

AT&T expects service to be restored by midnight tonight.

AT&T Network Engineers will be performing a planned maintenance on
Birmingham-Alabama Gigabit Access Router 3. The work will be performed on 6/18/2010,
during the maintenance window of 12:00AM-6:00AM, local time.
AT&T's goal is to successfully complete the task and restore service within the
stated time frame. 

We apologize for any inconvenience that you may experience.
We expect service to be impacted somewhat during the maintenance window.

AT&T Network Engineers will be performing an IOS upgrade.

We launched a new version of our web site!  Much of the information and functionality is the same, but we are updating the “look”.   Also, the new site is built on WordPress technology, which allows easy and quick updates of the site.   We are also tying the site in with Twitter.  You can follow us @netsolinc.   We will post updates of new products, specials, or events that may affect your service.  We hope you enjoy the new look and your feedback is welcomed.  If you have any issues with the site, please let us know!

We have one host down in our offsite datacenter.  It looks like a connectivity problem with the datacenter rather than a host issue.  We are working to resolve the issue.

Update 6:48pm – We have confirmed that it is yet another problem with a Cisco switch.  Datacenter engineers are working with Cisco now to resolve the issue.  Network connectivity has been intermittent since 6:37pm.

Update 8:40pm – Switching has been relatively stable for the last hour or so, but the datacenter engineers are continuing to research and diagnose the issue with Cisco TAC.

Update – Cisco identified a software bug.  They have applied updates to the equipment and the network is stable now.

Post Incident Report
Date / Time of Incident: 6/1/2010 6:44PM ET
Duration of Incident: 1hour 13minutes
Scope of Incident: Newark, Delaware Data Center Dedicated, Colocated and Cloud Customers

Description of Incident:
At 6:44PM ET one of the two redundant Cisco 6509 switches supporting the Dedicated, Colocated and Cloud Customers failed.  The second switch failed in the same manner when load from the first switch was re-routed to the second.  The end result was loss of connectivity for approximately 40 minutes.  As the switches recovered, latency lasted another 33 minutes.

Actions During Incident:
6:44 PM  - Network Operations and Network Engineering were immediately dispatched to investigate and resolve the issue.  A ticket was opened with the vendor of the switches (Cisco) and efforts to restore service were coordinated between the datacenter and Cisco.  

6:55 PM - After exhausting normal troubleshooting avenues, the Switch 1 is reloaded at console by Network Engineers

6:58 PM – Switch 2 primary supervisor engine fails to boot, secondary supervisor engine becomes active

7:03 PM – Switch 1 completes rebooting , routing protocols established, datacenter team monitors performance of switch

7:08 PM – Switch 2 Primary supervisor engine is manually reset, secondary supervisor engine becomes fully active

7:12 PM – Switch 1 primary supervisor engine fails, secondary supervisor begins to take over

7:20 PM – Switch 1 secondary supervisor takes over

7:37 PM – Switch 1 Primary supervisor engine pulled, engineers suspect faulty hardware

7:48 PM – Switch 2 Secondary Supervisor engine completes booting, begins forwarding traffic.  CPU on both switches remain elevated for 10 minutes while routing protocols are re-established

7:58 PM – Forwarding returns to normal, data delivered to Cisco for further analysis

Root Cause of Incident:
Cisco has identified a specific bug as the likely root cause that created a memory leak which triggered the event.  Still awaiting final confirmation from Cisco on the bug, but initial findings indicate software issues as the root.

Further Actions:
Based on an expected confirmation from Cisco on the bug, a software patch will be applied in a lab environment.  After successful application of the patch, Network Engineers will be applying the patch during a scheduled and announced maintenance window.

The datacenter Network Engineering team is looking into other architectural changes to add additional diversity to the customers that are currently supported by the current set of Cisco 6509s to further insolate customers from any future type of single core switching failure.

Our phone lines have all been ported over to voice over IP.   Preliminary testing looks good.  If you are interested in this type of system at your own office, please contact us.  It can save you a lot of money on your phone lines.

This morning, our phone lines are switching from traditional copper to voice over internet protocol (VOIP).   We are increasing the number of lines coming into our phone system to decrease busy signals.  The system will also allow us more flexibility with our system to allow us to serve you better.  During the switch, our office may be unreachable via phone for a short period of time.  If you have an issue, our technicians will be closely monitoring our support@internetpro.net email address. The window for this switch is scheduled from 8:30am to 10:30am CDT.

We have setup a new Twitter account for Network Solutions, Inc. You can follow it @netsolinc. Following that account will allow you to receive updates about new products and services as well as up to the minute network status updates. These updates will also be tied in to our new website.