2010-07-01
Abstract
Sender authentication is a hot topic in the world of email. It has a number of uses and a number of suggested uses. Which ones work in real life? Which ones don’t quite measure up? Can we use authentication to mitigate spoofing? Can we use it to guarantee authenticity? And how do we authenticate email, anyway? Terry Zink provides the answers to these questions and more, this month focusing on SPF.
Copyright © 2010 Virus Bulletin
In the first part of this article (see VB, June 2010, p.7), we introduced the concepts of SMTP, Internet headers, how spammers will try to spoof headers and how I want to hear from my friend Tony who has moved from Seattle to Sacramento. I also want to ensure that messages that come from Tony are not being faked by someone else.
Suppose before he left, Tony told me he would only send mail to me from Sacramento. If he travels out of state, he won’t send me any mail until he gets home. This way, if I ever get a letter from Tony, I only need to check to see where it came from (assume that the US Post Office stamps the letter with the city from which it originated). If it’s from Sacramento (indeed, if it’s Tony’s exact address) then I know it came from him.
We saw previously that receivers of email can use public DNS to look up the IP address corresponding to the sending domain and check to see if the sending IP matches it. They can also check to see if the reverse DNS record of the sending IP matches the envelope sender.
This is all based on guesswork. The Sender Policy Framework, or SPF, is an open standard that allows senders to define explicitly which IP addresses are allowed to send mail for a domain under their control. SPF is an authentication technology that uses sending domains and sending IP addresses to make decisions about authenticity.
In addition, one of the weaknesses of SMTP is that the sender can assign any email address as the Envelope sender and specify any other email address as the sender in the message headers. Thus, if a message gets past your spam filter and hits your inbox, you might be led to believe that the message is from someone who, in fact, did not send it. Most of today’s spam carries fake email addresses.
The current version of SPF – called SPFv1 or SPF Classic – protects the Envelope sender address, which is used for the delivery of messages. SPF allows an organization to specify where it will send mail from, and what receivers should do with the mail if they get a message purporting to be from them, but which doesn’t come from the IP addresses the organization has specified.
I should point out that when I say that SPF can authenticate a sender, what I mean is that it can validate that the email is sent from an IP address that the domain says is allowed to send mail. It does not necessarily follow that the user is authenticated. There are still cases where authenticated email can be malicious. An example is the case where a user’s credentials have been compromised and the thief uses that user’s account to send unwanted mail. Another example is the case of a computer being infected with a botnet and sending spam. In both cases, the mail will flow out from the proper mail server IPs and will be validated, but the user is not authenticated. For the purposes of our discussion, however, we will ignore such cases.
So how do we perform SPF checks in real life?
The SPF check is performed by the receiver. However, first the sender must publish their SPF records in DNS in the TXT record. The domain owner figures out all of the IP addresses that they know they send mail from and publishes this list in DNS. If the domain owner owns a block of IP addresses, that’s what they publish. The syntax is as follows (this is a simplified summary, for the full details see http://www.openspf.org/):
<version> <permitted sender> <mechanism>
The version is the version of SPF. For most domains, this is v=spf1.
The permitted sender is the list of records detailing who is allowed to send mail for the domain. These may be IP addresses, the A-record (i.e. look up the A-record for the sending domain), the MX-record (i.e. look up the MX-record for the sending domain), or it may redirect to another domain’s SPF record.
The mechanism specifies what to do with mail sent from IP addresses that do not fall into the range detailed in the permitted sender list. There are four qualifiers:
‘+’ Pass
‘-’ Hard fail
‘~’ Soft fail
‘?’ Neutral
The default mechanism is pass.
If the US Post Office had SPF records, then Tony’s SPF record might look like this:
v=USPostOffice2010 city:sacramento –all
From this, I can see that Tony only sends his letters from Sacramento. Only Tony is allowed to publish these post office SPF records. I should toss out anything that claims to come from him that does not come from Sacramento because the ‘-all’ indicates a hard fail. If I didn’t have Tony’s SPF record memorized, whenever I received a letter from anyone I’d have to phone up the US Post Office and ask for the SPF record. Of course, no such thing exists as US Post Office SPF records.
For microsoft.com, the SPF record is the following (snipped for clarity):
v=spf1 mx include:_spf-a.microsoft.com ~all
Larger organizations tend to have larger SPF records. Basically, this one states that any IP listed in the ip4 headings, and any IP listed in the SPF records for _spf-a.microsoft.com is a legitimate sender for microsoft.com. If any mail claiming to be from microsoft.com is sent from an IP range outside of those, it is a soft fail. Soft fails are interesting, as we will see later.
When a receiver gets a message, they check the SPF record of the domain in the envelope sender. They then check to see if the sending IP is listed in the SPF record. If so, the result is a pass. If not, the result is a fail.
SPF checks should be performed on the IP connecting into your organization. In other words, an email can take multiple hops to get to you, but you should perform an SPF check on the last hop before it reached you.
In the above diagram, the box represents a single organization. The email bounces around a few times, but IP 3 is the publicly facing IP and that is the one on which an SPF check is performed. While in this case, IP 1 and IP 2 may be internal IPs (as defined in RFC 1918), they could also be publicly facing IPs.
So why perform an SPF check on IP 3? There are two major reasons:
We saw in the last article that as each Mail Transfer Agent (MTA) passes the message along, it stamps a Received header on the message. In the above diagram, the Receiver stamped that it received the message from IP 3. The previous two MTAs also each stamped their own Received headers, receiving the message from MTAs 1 and 2, stamping IP 1 and 2, respectively. However, the Receiver has no way of validating whether these were stamped legitimately or if IP 3 stamped them all without them actually having gone through those relays. In other words, other than the one it stamped itself, the Receiver cannot tell whether any of the headers are valid or spoofed.
IPs that are internal are defined by RFC 1918. These are IPs in the range 192.168.0.0/16, 172.16.0.0/12, and 10.0.0.0/8. These are reused privately by organizations everywhere. They are not unique. In order for an SPF check to be useful, each organization must have its own set of unique IP addresses. If a receiver were to crawl through headers and end up with an IP in those IP ranges, then that would not be useful since they are used by different organizations all across the Internet.
Besides looking up the SPF records for the sending domain, comparing it to the transmitting IP and returning a pass or a fail, there are other possible results for the SPF check:
SPF hard fail – the transmitting IP is not permitted to send mail for that domain. This is used for domains that have knowledge of all of the outbound mail servers they might send from. If you know exactly where you’re sending from and who can send as you, SPF hard fail is the option you should let others know that you are using. A financial institution would be a good candidate for issuing SPF hard fail in its SPF records.
SPF soft fail – the transmitting IP does not meet a domain’s strict definition of legitimacy, but the domain cannot definitively classify the message as a forgery. Organizations that use this do not have all of their senders under their control.
SPF pass – the transmitting IP is permitted to send mail for that domain. If a sending mail passes an SPF check, then you know that the mail truly came from that domain.
SPF none – the domain does not publish SPF records. SPF is not required by SMTP, but its use is considered a best practice.
SPF neutral – the sending IP is outside of the domain’s SPF record, but the mail should be treated as if it had no SPF record. An example is forwarded mail – Gmail has an SPF neutral in its SPF record, but without it users would not be able to forward mail (i.e. from one account to the other) without incurring an SPF failure.
SPF error – the mail server received an error when looking up a domain’s SPF record.
SPF unknown – the mail server could not look up the domain’s SPF record, possibly because the domain’s SPF record is configured incorrectly.
Okay. So now we’ve got our mail and we’ve done an SPF check and got the result. What do we do with the mail?
If a sender passes the validation checks you can identify with confidence who is sending the message. However, a receiver should never assume that simply because someone has identified themselves, they can elevate that user’s level of trust.
If I get a letter from Tony in Sacramento that has been verified by the post office, I should trust that letter because I want to hear from Tony. If I get a letter from Bill in St. Louis that has been verified by the post office, I still have no reason to trust it, because I don’t know anyone named Bill. Bill gets no special treatment.
But in email, because I want to hear from Tony and I have validated that it is him, I could give the email special treatment. I could take all validated email from Tony and apply less filtering to it, or skip filtering altogether. If I validate mail from a trusted domain like PayPal, I might put it onto a fast track and skip filtering to avoid false positives from it. The rest of the email goes through the normal pipeline, subject to all of the filtering and possible delays that spam filtering incurs.
In other words, an SPF pass allows you to trust a certain class of user and be more aggressive on the rest of the mail flow.
While being more aggressive on the rest of the mail is a nice idea, it doesn’t work in practice. Spam filters are prone to false positives, but most spam filters are already reasonably accurate on spam. By becoming more aggressive, the mail that is already flagged as spam is flagged as ‘spammier’, but mail that is legitimate, but somewhat close to the spam threshold is now also flagged as spam. Spam that your filter wasn’t catching before still won’t be caught (at least most of it). The incremental catch rate of a more aggressive filter is not worth the trade-off of more false positives.
Action should only be taken on mail that passes an SPF check. If you want to implement a safe sender in your email, the only safe time to do so is when it passes an SPF check (we’ll get to other methods of validation in a future article). If you want mail to go through a separate lane for processing, it should only be done on mail that passes the check. The rest of the mail should be left alone, reverting to the default actions.
What about mail that fails an SPF check? What can you do?
The answer is that it depends. SPF hard fails generally allow the domain owner to specify what they want the receiver to do with mail when it fails a check. Financial institutions like Wells Fargo and PayPal often see their brands hijacked where spammers will attempt to send spoofed mail from e.g. [email protected] or [email protected]. Since it is relatively common for spammers to use the exact email domains of these institutions, upon determination of a hard fail these types of mails should be discarded.
SPF soft fails are a bit trickier. The recommendation is to accept the mail, but mark it. This could be used in a scoring engine and used as a weight in the spam filter. SPF soft fails are not definitive. Hotmail uses an SPF soft fail as part of its SPF record. If you forward a message from a Hotmail account to another account but retain the sender address, this will (soft) fail an SPF check because the sending IP will be different but the address is still the same. However, in this case you still want the mail, so while it might not be clean from a spam filter’s perspective, SPF soft fails alone are not enough to determine a message as spam.
While, theoretically, SPF hard fails should only be done when you want to discard mail with certainty, in practice, false positives occur. When someone is out on the road and connects to their mail server from a hotel or relays mail from an IP outside of their organization, SPF checks fail. The fact is that not everyone always accounts for their sending IPs. There almost needs to be yet another mechanism – hard fail for organizations that think that they know all of their sending IPs but for whom spoofing isn’t going to hurt their end-users, and super-hard fail for banks and organizations who have end-users that will suffer if they are tricked.
In my organization, a hard fail is assigned a heavier weight than a soft fail. Some organizations want to reject all mail that fails an SPF check, so that option is available to them.
Newsletters are an interesting case as they are especially prone to SPF soft fails. Many organizations outsource their mail campaigns to third-party services. These services send out mass mail to an organization’s subscribers and attach the organization’s name as the Envelope sender. Of course, the organization publishes SPF records and when the mail servers perform an SPF check, it fails. This is SPF working as intended. The organization could fix the problem by adding the third-party mailer’s IP address(es) to its SPF records. The drawback would be that the third-party mailer could conceivably use this to send out spam and damage the reputation of the organization.
Even though it’s easy for a spammer to spoof a domain and not use a brand’s real email address – for example, by using [email protected] instead of [email protected] (number 1 vs letter l) – in practice, spam filters get pretty good mileage out of using SPF checks in this manner. While it may be easy for a spammer to change the email address, many of them don’t and this SPF technique is very useful in catching those spammers.
One of the strengths of SPF is that it explicitly allows the sender to define the source IP addresses that are allowed to send mail for its organization. I don’t want any Tom, Dick or Harry to send as me, I only want me to send as me.
However, the reality is that not everybody has SPF records set up. It’s true that many large organizations do, especially in the United States. However, in other parts of the world (even Western Europe), SPF compliance is low. The receiver has no way to determine whether or not the purported sending organization does actually send mail from that IP. It’s a sad state of affairs.
Or is it?
One of the ways in which domains can be authenticated without SPF is by using an algorithm called ‘Best-Guess SPF’. This is intended to be a temporary measure until more domains come on board and start publishing their SPF records. The technique isn’t perfect, but it’s not bad, either. It works in the following manner:
Check the domain of the envelope sender. If it doesn’t publish SPF records, then check the MX-records and A-records of the sender’s domain. If the sending domain comes from the same range of IPs as the MX-record or A-record, then the sender has been authenticated.
Example 1 (using fictitious numbers)
Transmitting IP = 292.10.20.30
Envelope sender = [email protected]
A-record of tzink.com = 292.10.20.11
MX-record of lost.com = 292.10.20.16/28 (292.10.20.16 – 292.10.20.31)
Since the transmitting IP is within the range of the MX-records (an abnormally large MX-record, but hey, this example is fictitious), we have an authentication.
If that doesn’t work, get the reverse DNS of the sending IP. If it matches the domain of the envelope sender, then the sender has been authenticated.
Example 2
Transmitting IP = 292.10.20.32
Envelope sender = [email protected]
A-record of tzink.com = 292.10.20.11 -> No match
MX-record of lost.com = 292.10.20.16/28 (292.10.20.16 – 292.10.20.31) -> No match
Reverse DNS of 292.10.20.32 = tzink.com -> Match!
The reverse DNS name matches the name of the domain in the envelope sender, so the sender is authenticated.
Example 3
Transmitting IP = 282.10.20.32
Envelope sender = [email protected]
Reverse DNS of 292.10.20.32 = <no reverse DNS> -> No match
The reverse DNS name does not match the envelope sender, therefore there is no sender authentication.
If that doesn’t work, use a technique known as PTR zone. If the sender is a subdomain of the DNS PTR’s zone, then it is authenticated as if the sender comes from the zone itself.
Example 4
Transmitting IP = 282.10.20.32
Envelope sender = [email protected]
Reverse DNS of 292.10.20.32 = fantastic.tzink.com -> Close, but awesome.tzink.com is not a subdomain of fantastic.tzink.com.
Example 5
Transmitting IP = 282.10.20.32
Envelope sender = [email protected]
Reverse DNS of 292.10.20.32 =tzink.com -> Match because awesome.tzink.com is a subdomain of tzink.com
Gmail uses Best-Guess SPF, and using this extra bit of authentication allows it to authenticate almost twice as much mail as a standard SPF check (at least it did back in 2006 when first presented at the Conference on Email and Antispam). That’s actually pretty good. Best-Guess is non-standardized and specific to the implementation, but does appear to have some valid uses. As mentioned earlier, authentication in this manner allows the receiver to skip some processing on mail they want to receive.
SPF is a good framework for implementing sender authentication. It works for a number of reasons:
It is simple. Take the domain from the envelope sender, check its SPF record in DNS and see if the sending IP is in that range. That’s a pretty simple algorithm.
It allows strangers to set up their policies and strangers to look them up. I don’t need to know what Microsoft’s SPF record is, I can look it up in DNS. If I receive mail from someone I have never heard of before, I can check to see if they are coming from an authenticated source.
It allows you to authenticate senders you want to hear from. Since SPF specifically states how a check can be passed (namely, that an IP falls within the range), you can then choose to take action on those senders.
It allows you to reject mail from people who are spoofing another organization. This is an underappreciated tactic, but spam filters find great value in using SPF to discard phishing mails.
It allows organizations to tell you what to do with spoofed mail. This is really a follow-on from point 4, but nonetheless, if the cost to your organization’s user base is high if it is spoofed, then you want a more aggressive policy when someone impersonates your organization and a receiver detects it.
Yet for all of these strengths, there are still several questions: What are SPF’s drawbacks other than those we have outlined in this article? Can spammers get around SPF? If so, how? What is SenderID? How does that fit into things, and why was it even developed? Does it have any strengths that SPF doesn’t have?
These questions will be addressed in the next article.
