We all receive SPAM on a daily basis. It is unfortunately unavoidable. However, many of the aforementioned email services above have gotten pretty good at blocking a large percentage of it. DecaTech has an email filtering service called SpamAssassin in place that we rely on to cut down on SPAM as well. This is provided for free to all of our clients that utilize email service with us. This service does 2 things – thru complex algorithms it analyzes the email and assigns a “score” to it to identify the liklihood of it being SPAM. The higher the score, the more likely that it is SPAM. We’ve found (and its recommended by SpamAssassin) that any email with a score over 7.00 is most likely SPAM. The second thing it can optionally do is automatically delete any email that has higher score than a specified value. This is before the email is passed on, but it is permanent – an email detected as SPAM and automatically deleted is not recoverable.

When we forward on all email from a DecaTech email account to an external service, we are invariably also passing on SPAM. This SPAM does not originate or have anything to do with DecaTech’s webserver, but to the external email service, it appears like DecaTech is sending SPAM! To avoid getting black-listed (which usually leads to getting blocked) by companies like Google, Yahoo!, and Comcast, we need to step up our efforts to not pass on SPAM, whenever and however it can be helped.

Thus, we reach the DecaTech Email policy change:

If you have us forward email from a DecaTech email account, we must enable the SpamAssassin feature that deletes all detected SPAM. If you have any concerns about false positives – you can have us bump up the SPAM score a bit, but to no more than 8.0. A false positive is where an email is incorrectly marked as SPAM. You can check on this process now in two ways – look for emails that we have marked in the subject line with “**SPAM** (<score>)” or look at your regular email header.

For example, in Gmail, when viewing a specific email, click on the down arrow next to the “Reply” button and choose “Show original”. Look for a line in the email headers that looks like this:

X-Spam-Status: No, score=0.4 required=7.8 tests=AWL,BAYES_00,
	FH_DATE_PAST_20XX,HTML_MESSAGE autolearn=no version=3.2.4

The score in this example is 0.4, which basically means that the likelihood that this particular email is SPAM is extremely low.  You can also see that SpamAssassin was set to mark it as SPAM only if the value was higher than 7.8.

If you have any questions, or would like us to modify the SPAM detection value to something other than 7.0, please email DecaTech Support.

I’ve put a few specific blockers in place that have helped, and email is starting to catch up again to real-time activity.

If you are interested, this specific issue seems to be generating the majority of the attack:

http://www.symantec.com/connect/blogs/new-spam-attack-abusing-amazon-apple-twitter-email-notification

Also, our apologies for any confusion that the last couple of Blog notice emails sent out earlier this week may have caused. Both were blog posts from a month or more ago, but the outgoing notification messages were blocked due to a technical issue with the blog software.

The outage was a major power failure at the data center that the DecaTech server is housed at in Phoenix, AZ. The data center is designed to handle power outages, but this one went well above and beyond the capability of the data center, and let to an extended outage for our server and 100′s of others. Thanks to the diligence of our network provider, they were able to restore our server as rapidly as possible.

Here’s a technical break-down of what happened from their perspective, for those that are interested:

At approximately 8:00AM Central Time, utility power to our Phoenix data
center and the surrounding area was cut. The City of Phoenix has not yetinformed us of what caused this issue. Upon failure of utility power,backup generator fired-up as designed but the switching gear failed totransfer load to the backup power. Codero staff immediately contacted allof our power vendors as well as building maintenance since the transfergear and generator are not owned by Codero and are out of our physical control. Unfortunately, the electrical group which could manually transfer the load did not arrive before our UPS and battery loads diminished, and electrical service shutdown.

Upon arriving, faulty breakers had to be replaced before they were able to transfer the load to the backup power source. Shortly after this transfer, utility power was restored, and the PHX data center is currently back on utility power.

One of our primary PDU’s experienced issues as a result of the complete loss of power. This PDU services many customers, as well as many of the primary core network gear. Engineers immediately went to work on fixing the PDU issue.

We offer our sincerest apologies and our assurance that we are doing everything within our control to return services to normal for you.

If you have any further questions on this issue – please contact us via our Contact webpage.

Thru some automated scripts I have running, I was notified of the problem and temporarily resolved the problem around 2:45. After a few quick tests, the server appeared to be fully functional again. Unfortunately, the issue returned and the DNS service was down again, providing the same access issues as before. Our primary support was unable to look into the issue further for several hours. When they did, their server access was unfortunately blocked from their remote location, making it impossible for the issue to be immediately resolved. However, they were able to research the root cause of the DNS service issues, and relay this to our network provider. With their help, the problem was resolved at 8:55AM the next morning – March 2nd.

We apologize for this 19 hour outage and all of the inconveniences that it caused. We know that our clients rely on DecaTech to provide reliable hosting and email service – we are sorry for this interruption of service and the impact it had on your business and personal websites.

We have resolved the root issue that caused the problem and tested that future server reboots (planned or un-planned) no longer have the issue with DNS starting up correctly. We are also looking into additional options for backup server support so that we can more quickly resolve an issue like this in the future.

Email that was sent to your DecaTech hosted domain during the outage was unable to be delivered. Most servers automatically will attempt to resend email that can’t be delivered approximately every 4 hours for several days. As you then saw, you received these delayed email messages after 9AM when the DNS service was fully restored.

If you have any questions about the outage, please contact support@decatech.com.

Here’s a full disclosure on what happened:

• At 4:46AM on Tuesday, January 27th, our upstream network provider started rejecting all outgoing email at their relay server
• During the course of the day, a few clients notified DecaTech Support and expressed a concern that some outgoing emails were not being received
• DecaTech Support confirmed the issue in the late afternoon and determined that it was unable to resolve the issue on the server
• At 9:30PM the network provider was notified of the issue.
• For the next 3-4 hours DecaTech Support continued to research the issue and provided the network provider with very specific details
• at 11:30AM on Wednesday, January 28th, email stopped being rejected by the relay server and started going out
• Through-out the course of that day, the large queue of email was sent out. Outgoing email returned to real-time at approximately 5:00PM.
• DecaTech Support was notified by the network provider that the problem had been resolved on their end at around 4:45PM

Just to be clear, no outgoing email was lost or discarded – it was unfortunately delayed to varying degrees from Tuesday morning until Wednesday afternoon.

Moving forward and lessons learned:

Here at DecaTech, we’re not very happy with the responsiveness of our network provider and will be communicating with them in this regard. We want our hosting clients to expect and have excellent service and near zero interruptions of that service. That’s what you are paying for when you host with us.

DecaTech could have discovered the issue sooner, and at least communicated with the network provider earlier in the interruption of service to minimize the overall interval. We are working on creating a new automated script that can detect this problem if it ever occurs again.

One final note – when DecaTech Support was attempting to resolve the problem on our end, before it was determined that the issue lay with the network provider and their relay server, we decided to reboot our server. We take rebooting the server very seriously, as this causes a small window (less than 5 minutes) of interruption of all services – web, webmail, and email. We’re proud to report that the server had not been restarted previously for 554 days. One small bright spot.

As always, thank you for your patience and understanding during this interruption of service. We are sorry for any headaches and inconveniences it caused to our clients. Thank you for your continued business.

Comments and feedback are always welcome.

I’ve given as much info as I can provide from my end, and now I have to wait for a response from the network provider’s 2nd tier support. I hope to hear back good news and have the issue resolved in the next 5-7 hours.

All outgoing email is safely in the queue on the server, it has just been stuck there all day since the problem first arose. As soon as the network provider can resolve the connectivity issue, email should start flowing out.

As a side note, and during the efforts to get this resolved, I noticed that the server had not been rebooted in 554 days – that’s a lot of continuous uptime!

I’ll follow up with a final update as soon as this issue is completely resolved. As always, I’m very sorry for this inconvenience in delayed email service, and we at DecaTech appreciate your patience as we work to get it fixed.

Internal email (sent to other DecaTech clients, or DecaTech itself) is unaffected, along with incoming email.

We are working to resolve the problem, and will report status and when it has been successfully resolved via this blog.

However, there are some things that you can do, that can at least reduce the incoming SPAM to a tolerable level.

Part of the solution depends on what you use for your email client. We find that Outlook’s built-in spam filter works pretty well as a first line of defense. You also have SPAM protection from DecaTech on all of your email accounts with us.

Here’s what a  “marked” SPAM message will look like from DecaTech:

The email being marked as SPAM by DecaTech’s server will look like this on the Subject Line:

*****SPAM***** (12.0) The only formula for men that works

The first part is to uniquely identify it as spam.

The second part in parenthesis is the score of the SPAM – the higher it is, the more spam identifier features have been detected in the email

The third part is the regular subject of the email.

Currently, DecaTech’s servers are set at 6.2 as the SPAM value cut-off. If the score is higher than 6.2, it will be identified as SPAM. The

If you are comfortable with what you are seeing, you can request us to have the SPAM messages automatically deleted by the server so you will never see them. However, if you think that some SPAM identified messages are false positives, than you may not want to do this, as you’ll never receive that email. The choice is yours.

Just to be clear, the DecaTech server does not delete any of the SPAM it identifies by default, unless you ask us to have it do it.

There are filters in Outlook and DecaTech Webmail that you can setup by referring to their documentation so that they are automatically filtered out of your main Inbox as well.

Lastly, We recommend having your personal email go thru either a separate domain or thru one of the many free email services out there – Gmail is our favorite. Separation of business vs personal email can also help greatly in the battle with SPAM.

It appears that DecaTech Support has resolved the issue with sending email to AOL accounts thru the DecaTech server.

The solution is ironic and a little frustrating in that it involved disabling a spam flitering standard protocol that AOL championed to the world about 4 years ago, and that they now appear to have broken and violating with the way they have configured their email/web servers.

We are sorry that this issue took so long to resolve, but it took many hours of research and trial and error to find the solution.

Unfortunately, attempts to contact AOL both during the research effort and afterwards when a concrete solution was found, have so far proved to be futile. Decatech would like to pass on its findings to AOL but attempts to contact AOL’s postmaster have only resulted in empty auto-responder emails that say we’ll be contacted within 24-48 hours, and no contact ever comes.

If you have a choice, please look into alternatives to AOL – there are many better options in this day and age. If you have colleagues/friends that still use AOL, recommend that they look into switching to a free Gmail account, Yahoo, Hotmail, or one of countless other better alternatives.

If you have any questions related to this issue, please don’t hesitate to contact Decatech Support.

It has been reported from several sources that there is an issue with being able to send email to AOL accounts from DecaTech client email accounts.

There appears to be no issue with receiving email from AOL, which is normally more problematic, due to their aggresive black-listing policies.

We will be looking into this issue, and also contacting AOL to see if they can assist with getting this issue resolved as soon as possible for DecaTech clients.

If you have any questions regarding this issue, please contact DecaTech Support via our contact page.