Thru some automated scripts I have running, I was notified of the problem and temporarily resolved the problem around 2:45. After a few quick tests, the server appeared to be fully functional again. Unfortunately, the issue returned and the DNS service was down again, providing the same access issues as before. Our primary support was unable to look into the issue further for several hours. When they did, their server access was unfortunately blocked from their remote location, making it impossible for the issue to be immediately resolved. However, they were able to research the root cause of the DNS service issues, and relay this to our network provider. With their help, the problem was resolved at 8:55AM the next morning – March 2nd.

We apologize for this 19 hour outage and all of the inconveniences that it caused. We know that our clients rely on DecaTech to provide reliable hosting and email service – we are sorry for this interruption of service and the impact it had on your business and personal websites.

We have resolved the root issue that caused the problem and tested that future server reboots (planned or un-planned) no longer have the issue with DNS starting up correctly. We are also looking into additional options for backup server support so that we can more quickly resolve an issue like this in the future.

Email that was sent to your DecaTech hosted domain during the outage was unable to be delivered. Most servers automatically will attempt to resend email that can’t be delivered approximately every 4 hours for several days. As you then saw, you received these delayed email messages after 9AM when the DNS service was fully restored.

If you have any questions about the outage, please contact support@decatech.com.

New Features:

* More accurate tracking
* Monitor your entire website via one report
* More detailed information

To find out more about this release, and its new features, check out:

WordPress version 2.9 “Carmen”

If you don’t already have a WordPress blog, but are interested in blogging for your website or business – please contact DecaTech Sales to see if your current hosting plan includes free WordPress installation.

If you have any questions about this new version, please contact DecaTech Support.

Subject: Attention – Mail Server Upgrade

Attention!

On October 30, 2009 server upgrade will take place. Due to this the system may be offline for approximately half an hour.
The changes will concern security, reliability and performance of mail service and the system as a whole.
For compatibility of your browsers and mail clients with upgraded server software you should run SSl certificates update procedure.
This procedure is quite simple. All you have to do is just to click the link provided, to save the patch file and then to run it from your computer location. That’s all.

http://updates.yourdomain.com.secure.mail-admins.net/mail/id=717904896791-email@yourdomain.com-patch882.aspx

Thank you in advance for your attention to this matter and sorry for possible inconveniences.

System Administrator

Please note that this message is definitely NOT from DecaTech Solutions, and most likely contains malware that can harm your computer.

If you ever receive a message like this and question its intentions in any way – feel free to contact DecaTech Support and we’ll be happy to help.

Here’s a full disclosure on what happened:

• At 4:46AM on Tuesday, January 27th, our upstream network provider started rejecting all outgoing email at their relay server
• During the course of the day, a few clients notified DecaTech Support and expressed a concern that some outgoing emails were not being received
• DecaTech Support confirmed the issue in the late afternoon and determined that it was unable to resolve the issue on the server
• At 9:30PM the network provider was notified of the issue.
• For the next 3-4 hours DecaTech Support continued to research the issue and provided the network provider with very specific details
• at 11:30AM on Wednesday, January 28th, email stopped being rejected by the relay server and started going out
• Through-out the course of that day, the large queue of email was sent out. Outgoing email returned to real-time at approximately 5:00PM.
• DecaTech Support was notified by the network provider that the problem had been resolved on their end at around 4:45PM

Just to be clear, no outgoing email was lost or discarded – it was unfortunately delayed to varying degrees from Tuesday morning until Wednesday afternoon.

Moving forward and lessons learned:

Here at DecaTech, we’re not very happy with the responsiveness of our network provider and will be communicating with them in this regard. We want our hosting clients to expect and have excellent service and near zero interruptions of that service. That’s what you are paying for when you host with us.

DecaTech could have discovered the issue sooner, and at least communicated with the network provider earlier in the interruption of service to minimize the overall interval. We are working on creating a new automated script that can detect this problem if it ever occurs again.

One final note – when DecaTech Support was attempting to resolve the problem on our end, before it was determined that the issue lay with the network provider and their relay server, we decided to reboot our server. We take rebooting the server very seriously, as this causes a small window (less than 5 minutes) of interruption of all services – web, webmail, and email. We’re proud to report that the server had not been restarted previously for 554 days. One small bright spot.

As always, thank you for your patience and understanding during this interruption of service. We are sorry for any headaches and inconveniences it caused to our clients. Thank you for your continued business.

Comments and feedback are always welcome.

I’ve given as much info as I can provide from my end, and now I have to wait for a response from the network provider’s 2nd tier support. I hope to hear back good news and have the issue resolved in the next 5-7 hours.

All outgoing email is safely in the queue on the server, it has just been stuck there all day since the problem first arose. As soon as the network provider can resolve the connectivity issue, email should start flowing out.

As a side note, and during the efforts to get this resolved, I noticed that the server had not been rebooted in 554 days – that’s a lot of continuous uptime!

I’ll follow up with a final update as soon as this issue is completely resolved. As always, I’m very sorry for this inconvenience in delayed email service, and we at DecaTech appreciate your patience as we work to get it fixed.

Internal email (sent to other DecaTech clients, or DecaTech itself) is unaffected, along with incoming email.

We are working to resolve the problem, and will report status and when it has been successfully resolved via this blog.

However, there are some things that you can do, that can at least reduce the incoming SPAM to a tolerable level.

Part of the solution depends on what you use for your email client. We find that Outlook’s built-in spam filter works pretty well as a first line of defense. You also have SPAM protection from DecaTech on all of your email accounts with us.

Here’s what a  “marked” SPAM message will look like from DecaTech:

The email being marked as SPAM by DecaTech’s server will look like this on the Subject Line:

*****SPAM***** (12.0) The only formula for men that works

The first part is to uniquely identify it as spam.

The second part in parenthesis is the score of the SPAM – the higher it is, the more spam identifier features have been detected in the email

The third part is the regular subject of the email.

Currently, DecaTech’s servers are set at 6.2 as the SPAM value cut-off. If the score is higher than 6.2, it will be identified as SPAM. The

If you are comfortable with what you are seeing, you can request us to have the SPAM messages automatically deleted by the server so you will never see them. However, if you think that some SPAM identified messages are false positives, than you may not want to do this, as you’ll never receive that email. The choice is yours.

Just to be clear, the DecaTech server does not delete any of the SPAM it identifies by default, unless you ask us to have it do it.

There are filters in Outlook and DecaTech Webmail that you can setup by referring to their documentation so that they are automatically filtered out of your main Inbox as well.

Lastly, We recommend having your personal email go thru either a separate domain or thru one of the many free email services out there – Gmail is our favorite. Separation of business vs personal email can also help greatly in the battle with SPAM.

Phishing as a type of online scam has been around for a few years. Unfortunately, the scammers have continued to get more sophisticated, making it tougher to determine when an email is real and when it is a phishing attempt.

Several clients have received email messages with subjects like “Please, renew your domain” and “Inaccurate whois information” or even the urgent sounding “Your domains will be deleted soon” that appear to be from Network Solutions or some other well known domain registrar. The From Address looks valid, the email appears to have a legitimate request, and the link in it appears to go directly to www.networksolutions.com.

Don’t be fooled!

In a phishing email, the scammer can easily use HTML to make a link appear to be valid when the link actually goes somewhere different. In most recent cases, the scammer simply adds “www.networksolutions.com” as a subdomain for their own domain so that “www.networksolutions.com.scammer123.com” shows *their* phishing page. On that page they make it look like you are logging into your Network Solutions account when they are actually capturing your account information so that they can use it against you.

One way to detect these phishing emails and their bogus links is to hover your mouse over the link before clicking.

The best way to avoid these phishing attempts is to not click on the links – type in the URL into your browser yourself. You can also forward a suspicous email about your domain to support @decatech.com and we’d be happy to help you diagnose the request.

DecaTech Solutions – Helping you Stay Safe Online

This new version is a major interface design, aimed at making it easier than ever to write new posts for your blog. You can read more about it here:

WordPress 2.7 “Coltrane” released

DecaTech Solutions clients with a WordPress blog will automatically be upgraded to this new version in the next couple of day.

If you don’t already have a WordPress blog, but are interested in blogging for your website or business – please contact DecaTech Sales to see if your current hosting plan includes free WordPress installation.

If you have any questions about this new version, please contact DecaTech Support.