Comments on: My Cari.net dedicated server is down..and so is cari.net! http://webolio.wordpress.com/2008/03/13/my-carinet-dedicated-server-is-downand-so-is-carinet/ Fri, 06 Nov 2009 20:12:44 +0000 http://wordpress.com/ hourly 1 By: Shawn http://webolio.wordpress.com/2008/03/13/my-carinet-dedicated-server-is-downand-so-is-carinet/#comment-51 Shawn Wed, 04 Jun 2008 18:15:21 +0000 http://webolio.wordpress.com/?p=17#comment-51 This is posted under out "network status" page http://www.cari.net/network-status.html?netstat_month=3&netstat_year=2008&netstat_day=false&netstat_start=0 Datacenters: C2, C3, C4, C5 Date & Time (PDT): Thursday, Mar 13 2008, 08:51 – Friday, Mar 14 2008, 10:10 Effect on network continuity: --A variety of connectivity problems ranging from DNS to direct Internet connectivity issues. Details At 8:51am Cari.net’s network began experiencing significant routing and BGP issues. The problems continued until 10:10am, during which time customers experienced a variety of connectivity problems ranging from DNS to direct Internet connectivity issues. We apologize for the outage and any inconvenience it may have caused. A full scale inquiry is being conducted and further details will be provided within the next 2 hours. If you are still experiencing connectivity issues, please call our 24/7/365 Support Center or open a trouble ticket. -------------------------------------------------------------------------------- The following information is data that has been compiled and it being provided as our 2 hour post incident update. Cari.net management and networking teams are working to evaluate this mornings connectivity issues in an effort to adequately explain and prevent further occurrences. Below is our 2-hour post incident update. The information provided is based on “first response” review of the situation. Another update will be provided before close of business today with more a more in-depth and definitive review of the situation. ***************************************** Broadcast storm of undetermined origin caused link flapping which in turn caused HSRP and spanning tree failures. The broadcast storm apparently began in C2 data center, disrupting traffic on key corporate vlans as well as hosted servers. The C2 core router’s CPU became overloaded and inter-data center links were non-responsive, causing STP recalculations and HSRP failures. Key corporate infrastructure became inaccessible as multiple routers attempted to take over (or relinquish) gateway IPs as spanning tree calculated switching paths appeared and disappeared. The C2 router shares switching infrastructure with the C3 core and the initial state of the data center interconnections had most traffic passing through the C5 data center. The broadcast storm cascaded through both the primary and backup C5 distribution networks, leaving access switches with no egress. The broadcast storm propagated through the shared switching infrastructure of the C3 data center facility. Both primary and redundant customer colocation access routers were affected and the storm propagated to the customer access switches. As a result, many customer access devices (in the colocation cabinets) were left in a non-functioning state and required a reboot to restore services. Cisco engineers are on site to determine the root cause of the issue. In the interim we have taken the steps to deploy additional equipment and to remove certain HSRP and redundant switch paths to reduce the severity of link flapping in 100% resolution is proven. ***************************************** As stated above, all resources are focused on this critical situation and Cari.net is leveraging other business relationships to have third party visibility and oversight into the issue resolution process. We thank you for your continued patronage as we work to provide 100% resolution to this issue. This is posted under out “network status” page
http://www.cari.net/network-status.html?netstat_month=3&netstat_year=2008&netstat_day=false&netstat_start=0

Datacenters: C2, C3, C4, C5
Date & Time (PDT): Thursday, Mar 13 2008, 08:51 – Friday, Mar 14 2008, 10:10

Effect on network continuity: –A variety of connectivity problems ranging from DNS to direct Internet connectivity issues.

Details
At 8:51am Cari.net’s network began experiencing significant routing and BGP issues. The problems continued until 10:10am, during which time customers experienced a variety of connectivity problems ranging from DNS to direct Internet connectivity issues.

We apologize for the outage and any inconvenience it may have caused. A full scale inquiry is being conducted and further details will be provided within the next 2 hours.

If you are still experiencing connectivity issues, please call our 24/7/365 Support Center or open a trouble ticket.

——————————————————————————–

The following information is data that has been compiled and it being provided as our 2 hour post incident update. Cari.net management and networking teams are working to evaluate this mornings connectivity issues in an effort to adequately explain and prevent further occurrences. Below is our 2-hour post incident update. The information provided is based on “first response” review of the situation. Another update will be provided before close of business today with more a more in-depth and definitive review of the situation.

*****************************************

Broadcast storm of undetermined origin caused link flapping which in turn caused HSRP and spanning tree failures. The broadcast storm apparently began in C2 data center, disrupting traffic on key corporate vlans as well as hosted servers. The C2 core router’s CPU became overloaded and inter-data center links were non-responsive, causing STP recalculations and HSRP failures. Key corporate infrastructure became inaccessible as multiple routers attempted to take over (or relinquish) gateway IPs as spanning tree calculated switching paths appeared and disappeared. The C2 router shares switching infrastructure with the C3 core and the initial state of the data center interconnections had most traffic passing through the C5 data center. The broadcast storm cascaded through both the primary and backup C5 distribution networks, leaving access switches with no egress. The broadcast storm propagated through the shared switching infrastructure of the C3 data center facility. Both primary and redundant customer colocation access routers were affected and the storm propagated to the customer access switches. As a result, many customer access devices (in the colocation cabinets) were left in a non-functioning state and required a reboot to restore services. Cisco engineers are on site to determine the root cause of the issue. In the interim we have taken the steps to deploy additional equipment and to remove certain HSRP and redundant switch paths to reduce the severity of link flapping in 100% resolution is proven.

*****************************************

As stated above, all resources are focused on this critical situation and Cari.net is leveraging other business relationships to have third party visibility and oversight into the issue resolution process. We thank you for your continued patronage as we work to provide 100% resolution to this issue.

]]>