Server Outage – apology and explanation
March 3, 2010 – 2:06 amAt approximately 2:00PM MT on Tuesday, March 1st… our server stopped being able to respond to requests for all domains that are hosted on it. Basically, the DNS service was down so access to domains for web and mail requests was down. The root cause was an unexpected reboot of the server by our network access provider at 1:53PM. The server had not been (nor had it needed to be) rebooted in over 4 months, so the behavior of the DNS service crashing on the server restart was an unexpected issue.
Thru some automated scripts I have running, I was notified of the problem and temporarily resolved the problem around 2:45. After a few quick tests, the server appeared to be fully functional again. Unfortunately, the issue returned and the DNS service was down again, providing the same access issues as before. Our primary support was unable to look into the issue further for several hours. When they did, their server access was unfortunately blocked from their remote location, making it impossible for the issue to be immediately resolved. However, they were able to research the root cause of the DNS service issues, and relay this to our network provider. With their help, the problem was resolved at 8:55AM the next morning – March 2nd.
We apologize for this 19 hour outage and all of the inconveniences that it caused. We know that our clients rely on DecaTech to provide reliable hosting and email service – we are sorry for this interruption of service and the impact it had on your business and personal websites.
We have resolved the root issue that caused the problem and tested that future server reboots (planned or un-planned) no longer have the issue with DNS starting up correctly. We are also looking into additional options for backup server support so that we can more quickly resolve an issue like this in the future.
Email that was sent to your DecaTech hosted domain during the outage was unable to be delivered. Most servers automatically will attempt to resend email that can’t be delivered approximately every 4 hours for several days. As you then saw, you received these delayed email messages after 9AM when the DNS service was fully restored.
If you have any questions about the outage, please contact support@decatech.com.