Email sender authentication: advantages and shortcomings

The severity of the spam and malware calamity should not be new to readers of this publication; neither should the close relationship between the two, nor the difficulty in combating them. For new readers, things can be summed up as follows: the situation is perilous, spam and malware go hand in hand, and defence is difficult and expensive. It often seems we are in an arms race, with each side trying to out-gun their opponent. Although we all secretly hope for and seek the magic weapon that will obliterate the problem once and for all, the reality is such a weapon does not currently exist, and we still rely on our tried and tested methods of defence to help us survive.

During the last few years we have seen the emergence of several technologies commonly referred to as ‘email authentication’. Email authentication attempts to verify and certify the authenticity of an email. Email authentication is usually divided into two categories [1]:

This paper will focus mainly on sender authentication.

In reality it is hard to find a multi-purpose tool that is the best at everything. In order to use a tool effectively, it is essential to understand its strengths and weaknesses. The same applies to email authentication. In order to use it effectively, it is necessary to understand what it can do well, what it can’t do very well, and what it can’t do at all.

To understand properly the best application of email authentication technologies let’s begin with their deficiencies:

These deficiencies have been the cause of heavy criticism from some in the internet community. Several have even labelled authentication technologies as ‘useless’. We disagree. If used properly, email authentication can help reduce (not eliminate) spam.

So how do you use email authentication properly? You use it to help establish trust. For example, sender authentication helps answer the questions ‘is this email coming from the place it should come from?’ If the sender authentication fails, we have reason to doubt the legitimacy of the email. But if the authentication is successful it won’t automatically mean the email is legitimate.

In other words, failed authentication does not make an email spam and successful authentication does not make it ham. This is the area where many have problems understanding the proper use of email authentication. They erroneously believe that authentication by default equates to legitimacy. To their credit, spammers were quick to point out this flawed idea. Unfortunately, many who made this mistake were quick to blame the technology rather than their own misunderstanding of the technology for its failures and precipitately dubbed email authentication useless.

So what does email authentication do for us? Let’s begin with sender authentication in the form of SPF and its cousin SenderID.

SPF publishes rules via DNS records about which SMTP servers are allowed to send email for a particular domain. If we receive an email from a server that does not conform to the published rules, the possibility of header forgery for that email increases. SPF (if not set up too broadly) effectively reduces the number of SMTP servers that can send email for a given domain from hundreds of millions to a mere handful. This can help reduce the impact of botnets and help catch phishing attacks.

Message authentication can also help authenticate email by providing a verifiable signature for each email message. This helps us determine whether the message was signed by a valid server and whether the message has arrived unmodified.

One major shortcoming of sender authentication technologies is that they are not currently mandated. Not all domains have deployed these technologies and they don’t enjoy broad deployment. Politics have also played a role, with many early adopters taking sides with one technology or the other.

During the MIT Spam Conference 2007 [2] we presented a technique similar to SPF that uses existing DNS records to authenticate email. We called this technique ‘Relay Detection’ [3]. It works under the assumption that legitimate email is rarely relayed these days, and that spammers continue to relay spam, mostly through botnets. In other words, a Hotmail user will send email through Hotmail servers while a spammer claiming to come from Hotmail will use an open relay or a machine in a botnet to send the email.

The first step in Relay Detection is to verify the HELO identity of the sender SMTP, either during the SMTP transmission or through the correct Received header (see Figure 1). The SMTP standards indicate the sender SMTP should send its Fully Qualified Domain Name (FQDN). Therefore we verify the validity of the HELO identity as follows:

In our preliminary testing, these simple tests were capable of detecting over 60% of spam with less than 0.35% false positives. Although these numbers may not be terribly impressive, they can be extremely useful if these checks are implemented at the SMTP level before the email is even received.

The second step in Relay Detection is to verify the second command in the SMTP transmission: MAIL FROM (see Figure 2). If done after the SMTP transmission, you can use the Return-Path header. The main goal of this test will be to establish through DNS records a relationship between the domain information given in the HELO command with the domain in the MAIL FROM address. You can verify the validity of the MAIL FROM address as follows:

In our preliminary testing we were able to identify 78% of the spam messages not flagged by the Sender SMTP Verification tests with roughly a 0.5% false positive rate. Overall, these two forms of verification flagged over 90% of spam with a less than 1% false positive rate.

Be aware that the tests and their order are not arbitrary. They are designed to work for nearly all email configurations, including domains in virtual hosting environments.

The only cases where these tests fail is when headers are forged or email is relayed (after all, that’s what they are looking for). However, these tests are usually very fast and consume very little processing time, with most of the time being spent waiting for DNS queries.

It is important to note that sender authentication technologies have been adopted by many spammers. This greatly reduces the efficacy of email authentication as a spam-fighting tool.

Many who have encountered this unfortunate fact have concluded that these technologies simply ‘don’t work’ and are not worth the effort. Luckily, email authentication is still helpful in the fight against spam. Just because a screwdriver doesn’t make a good hammer doesn’t mean it should be thrown out.

As we mentioned earlier, email authentication helps the initial establishment of trust. Sender authentication in any or all of its flavours does one thing very well: it reduces the number of servers that can legitimately send email for a domain to a privileged few.

If used properly, sender authentication prevents header forgeries, forces spammers to be more honest and pushes them to start using their own domains. This has important positive implications for mitigating phishing attacks. For example, a scam artist trying to impersonate a bank (such as Bank of America) cannot simply forge the email headers any more.

In order to bypass these email authentication techniques the spammer would have to:

Not only does this start to affect the economics of spam, it also plants the seed of distrust for the receiver. An astute recipient receiving an email from ‘Bank of America Support’ <[email protected]> may wonder why the email did not originate from bankofamerica.com. Not all email users may be so savvy but anti-phishing techniques like those used by modern browsers could also help diffuse the attack.

Although our research may not be seen as revolutionary, it has helped us confirm several things:

In continuing with our research we may investigate further the relationship that could be established between sender identification and domain trust.

With sender authentication technologies helping verify the source of incoming email, we can more fully develop trust relationships between domains.

With the help of other spam-filtering technologies and the confidence that an email is coming from a verified source, it should be possible to measure a domain’s tendency to spew spam. Those measurements could be calculated automatically from incoming email and could provide another measure to help determine the probability of a message being spam. Even though several companies specialize in the analysis of spam for the purposes of publishing blacklists and such, this technique may improve blacklist and whitelist efficiency by ensuring the lists are relevant to the spam and legitimate email received by a particular host.

Another possible direction for our research is to use sender authentication as a means of prioritizing spam filtering, virus scanning and delivery queues. Along with the domain spam measurements mentioned above, incoming SMTP connections could be prioritized and handled appropriately, allowing efficient routing. For example, an untrusted domain could be greeted by a tar pit in an effort to slow the spammer and wait for the sender to give up. At the same time messages from another domain with a good ham record but poor virus record could immediately be sent through a very thorough virus scanner. And, in some cases, it may even be possible that some domains could bypass spam filtering and virus scanning altogether and be queued for immediate delivery.

These ideas are still in their early infancy. We are barely exploring the many possibilities provided by sender authentication. In either case, we feel confident that email authentication (both sender authentication and message authentication) can help reduce email if it is used as a tool for establishing trust and not mistaken as a complete spam solution in and of itself.