Server outage – Power issues at Network Facility
March 15, 2010 – 3:47 pmFrom approximately 7:15AM MDT until 9:40AM on the morning of March 15th, DecaTech clients experienced an interruption in web and mail service.
The outage was a major power failure at the data center that the DecaTech server is housed at in Phoenix, AZ. The data center is designed to handle power outages, but this one went well above and beyond the capability of the data center, and let to an extended outage for our server and 100′s of others. Thanks to the diligence of our network provider, they were able to restore our server as rapidly as possible.
Here’s a technical break-down of what happened from their perspective, for those that are interested:
At approximately 8:00AM Central Time, utility power to our Phoenix data
center and the surrounding area was cut. The City of Phoenix has not yetinformed us of what caused this issue. Upon failure of utility power,backup generator fired-up as designed but the switching gear failed totransfer load to the backup power. Codero staff immediately contacted allof our power vendors as well as building maintenance since the transfergear and generator are not owned by Codero and are out of our physical control. Unfortunately, the electrical group which could manually transfer the load did not arrive before our UPS and battery loads diminished, and electrical service shutdown.Upon arriving, faulty breakers had to be replaced before they were able to transfer the load to the backup power source. Shortly after this transfer, utility power was restored, and the PHX data center is currently back on utility power.
One of our primary PDU’s experienced issues as a result of the complete loss of power. This PDU services many customers, as well as many of the primary core network gear. Engineers immediately went to work on fixing the PDU issue.
We offer our sincerest apologies and our assurance that we are doing everything within our control to return services to normal for you.
If you have any further questions on this issue – please contact us via our Contact webpage.