CTO Tech Blog

Wouldn’t it be great if all unsolicited commercial email, or spam as it’s more commonly known, came with a tag in the subject line that identified it as spam?  Not surprisingly, the people who send spam don’t see the value in that idea.  The job of defending networks and users against spam is on the shoulders of network administrators and users.  Understanding what defences are available and how they work will help users and network administrators choose the best solution for their environment.

Spam filtering can be done in any of four points in the path of an email; as it is sent, before it reaches the destination server, on the destination server, or at the end client.  Checking for spam as it is sent seems, on the outside, to be futile since someone intending to broadcast spam would certainly not do it.  In fact, a large portion of spam is sent using malware that is unknowingly installed on end user hosts.  Checking for spam leaving a host would identify that there is a problem which could then be corrected using malware tools.  Companies that host their own mail servers, as well as many hosted servers check email for spam as it arrives or while it is being processed by the mail server.  Filtering spam before it arrives at the server reduces network traffic and isolates the server from malware that may be contained in spam, but often limits the ability for users to allow email that may appear to be spam from people or businesses that are known.  This is called white listing a sender or email domain.  Moving the spam filter to the mail server often adds simpler configuration tools to allow administrators or users to adjust the white list (desired) and blacklist (malicious) settings either on a global or per user basis.  Client based filtering allows users to individually decide how to identify spam and how to act on each message based on the rules that they configure.  Any or all of these types of filtering can be used depending on how much control a network administrator wants to retain or offload to users and most solutions are a hybrid mix.

Regardless of what stage on the email flow a filter is testing for spam, the same criteria can be used.  Most spam filters use a number of tests in a certain order with specific settings for action and logging for each test.  The best filter will allow all desirable mail through and block all spam, but since the spammers are constantly fighting the filters, new rules must constantly be implemented.  Having a subscription with the spam filter vendor allows the filter to be updated with defences against new spamming methods very soon after they are discovered.  Here are some of the more common methods included in spam filters to detect and protect against spam.

Heuristics/Bayesian analysis – These intelligent filters learn and use statistics to determine if a message is spam.

Reputation – Most anti-spam vendors keep databases of known spammers and tag any mail from their domain or IP address as spam.

Phishing – The scan engine looks for links in emails to known phishing sites and tags them.

RBL – Realtime or Relay blacklist – These are third party lists of reported spammers on the Internet.  These blacklists often list IPs and domains that have temporary issues with malware, or are part of a large range of addresses.

Header checking – The scan engine compares the SMTP and MIME email addresses in the message header to make sure they match as well as other header anomalies.  Email clients set the MIME address, while the server sets the SMTP address.  Spammers use this to mask the origin of an email.

Directory harvesting – Emails sent to multiple non-existent accounts in an email domain are marked as spam.  The term directory harvesting is used because by sending large lists of names to a mail server, most will bounce back as failed, but some may not.  Eventually the sender can, by process of elimination, build a list of valid email addresses on the domain.

SPF – Sender Policy Framework – A relatively new DNS record type that defines domain names and hosts that are authorized to send email on a given domain.  Although not widely used initially this method of protecting a domain has become more common.

rDNS – Reverse Domain Name Service – Where DNS takes a domain name and translates it to an IP address, rDNS looks at the IP address and ensures that the domain name in the email header matches the domain name that points to the IP.  This identifies servers that are relaying mail for other, typically unauthorized, domains.

Blacklist – A list on the anti-spam server that is generated automatically or by user intervention that identifies domains or IP addresses that users have reported as spam.

Keyword checking – A list generated automatically or by user intervention or words that are in undesirable emails.  These words could be profanity, pornographic, pharmaceutical, or any other s.

Regardless of which type of spam filtering is used and which tests are implemented, all but the most expensive appliances require configuration and some amount of learning.  Don’t expect a product to stop all spam right out of the box.  With some adjustments and ongoing updates they are great tools to keep users productive.

Email is a wonderful tool for communication, both at work and at home.  It's almost as common today to ask for someone's email address as their phone number.  Unfortunately most, if not all, of us have received some form of spam in our mailboxes as well as our intended communications.

Spam is officially known as unsolicited commercial email or UCE.  The more common term "spam" was reportedly borrowed from a Monty Pithon's Flying Circus comedy sketch of the same name.  The connection presumably being the annoying nature and apparent unavoidability of UCE and the similar use of the popular Hormel Foods canned ham in the sketch.  Unlike the food product, email spam comes in a variety of flavours.  Commonly we see prosthetic and drug advertising emails that use constantly changing techniques to attempt to thwart spam filters.  These messages frequently have return email addresses that are non existent or that have been hijacked from some innocent party (sender spoofing).  I'll walk through some methods of sending spam and discuss some techniques to help reduce the amount of spam you receive.

Since spammers, the people who originate spam email, need to protect themselves from being identified and blocked by intended recipients they have had to find ways to send a large number of emails from various locations in a very short period of time.  The most common method of doing this is to use an open mail relay somewhere on the Internet.  An open relay is a mail server that will forward email from anyone (or anyone who knows how to get around security measures) to any potential recipient on the Internet.   In the early days of spamming the sender would usually create a new email account on a large free mail hosting service such as Yahoo! or AOL and send emails to a large list of recipients.  As the hosting companies clamped down on this type of activity they began to use complex scripts with multiple open relay servers and large lists of recipients.  Today most commercial email providers block relaying, but many smaller companies who host their own email servers often don't have the technical skills required to turn off relaying while still allowing their users to send and receive emails that they need to.  The lists of email addresses that spammers send to are created using various methods.  Some companies sell lists of email addresses gathered from their web site.  Since email is sent unencrypted over the Internet, the addresses can often be gathered by "listening" to traffic between servers.  Sometimes they will even send email to random names in an email domain (a domain is the part after the @ in an email address) and see which ones don't bounce back to them (directory harvesting).

A few methods to help avoid spam are to not enter your email address in web forms or use it to enter draws.  In cases where you need to do this, consider creating a "spam account" with one of the online email providers like Yahoo! or GMail.  Use this account when you're dealing with companies on the web that you don't know.  If your Internet provider offers a spam filtering solution it may be worth subscribing to it.  On a larger network where email is hosted in house, an anti spam appliance or dedicated server software are the best defence as they offload the traffic from the mail server.  Some PC based spam filters work well, but they usually need a subscription to keep up to date so they can continue to work effectively.  Spammers are constantly changing their methods so new defences need to be defined and applied.  Junk mailboxes and blacklists that are built into applications such as Microsoft Outlook also help, but they tend to rely on filtering by the sender and these days the sender is changed with every mailing that goes out.

Just make sure you put my email address on your white list.