Outgoing Email Issue – RESOLVED
January 29, 2009 – 7:21 amThe issue with outgoing email has been resolved.
Here’s a full disclosure on what happened:
- At 4:46AM on Tuesday, January 27th, our upstream network provider started rejecting all outgoing email at their relay server
- During the course of the day, a few clients notified DecaTech Support and expressed a concern that some outgoing emails were not being received
- DecaTech Support confirmed the issue in the late afternoon and determined that it was unable to resolve the issue on the server
- At 9:30PM the network provider was notified of the issue.
- For the next 3-4 hours DecaTech Support continued to research the issue and provided the network provider with very specific details
- at 11:30AM on Wednesday, January 28th, email stopped being rejected by the relay server and started going out
- Through-out the course of that day, the large queue of email was sent out. Outgoing email returned to real-time at approximately 5:00PM.
- DecaTech Support was notified by the network provider that the problem had been resolved on their end at around 4:45PM
Just to be clear, no outgoing email was lost or discarded – it was unfortunately delayed to varying degrees from Tuesday morning until Wednesday afternoon.
Moving forward and lessons learned:
Here at DecaTech, we’re not very happy with the responsiveness of our network provider and will be communicating with them in this regard. We want our hosting clients to expect and have excellent service and near zero interruptions of that service. That’s what you are paying for when you host with us.
DecaTech could have discovered the issue sooner, and at least communicated with the network provider earlier in the interruption of service to minimize the overall interval. We are working on creating a new automated script that can detect this problem if it ever occurs again.
One final note – when DecaTech Support was attempting to resolve the problem on our end, before it was determined that the issue lay with the network provider and their relay server, we decided to reboot our server. We take rebooting the server very seriously, as this causes a small window (less than 5 minutes) of interruption of all services – web, webmail, and email. We’re proud to report that the server had not been restarted previously for 554 days. One small bright spot.
As always, thank you for your patience and understanding during this interruption of service. We are sorry for any headaches and inconveniences it caused to our clients. Thank you for your continued business.
Comments and feedback are always welcome.