Thread Spammer Club

Post Reply New Topic

How Spam Works

by Marshall Brain
Print
Cite This!
Close
Please copy/paste the following text to properly cite this HowStuffWorks article:

Brain, Marshall. "How Spam Works" 25 September 2003. HowStuffWorks.com. <http://computer.howstuffworks.com/internet/basics/spam.htm> 08 August 2011.
Cite
Feedback
39
Need tech help?
Check out our how-to tech articles.

Inside this Article

1. Introduction to How Spam Works
2. How Spammers Get Addresses
3. The Big Spamming Companies

4. How to Stop Spam
5. Lots More Information
6. See all Internet Basics articles

More on E-mail

* How E-mail Works
* How Gmail Works
* E-mail Marketing
* E-mail Scams
* Can the government read your e-mails?

Most of us get spam every day. Some of us get a little, and some of us get a lot, but if you have an e-mail account it is always there. For example, this morning, here's one that came to my inbox:

Subject: Adobe

Suppose we tell you that you could really lose up to 82% of your unwanted body fat and keep it off in just a few months, would you be interested? We certainly hope so! Please visit our web site - Click here!

Obviously this is spam, yet it made it through the spam filters and I opened it because the subject line made it unknowable whether it was spam or not. ­

Spam is incredibly annoying, especially in large quantities. If you have a public e-mail address you can receive hundreds of spam messages for every legitimate message that arrives. Even with good filters, some of the spam makes it through. And filters can sometimes delete messages that you really do want to receive. Spam is free speech run amok. ­

­ Where does all of this spam e-mail (also known as "unsolicited commercial e-mail") come from? Why is there so much of it? Is there any way to stop it? In this article, we will answer these questions and many others as we take a dive into the sea of spam.

Spam is a huge problem for anyone who gets e-mail. According to Business Week magazine:

In a single day in May [2003], No. 1 Internet service provider AOL Time Warner (AOL ) blocked 2 billion spam messages -- 88 per subscriber -- from hitting its customers' e-mail accounts. Microsoft (MSFT), which operates No. 2 Internet service provider MSN plus e-mail service Hotmail, says it blocks an average of 2.4 billion spams per day. According to research firm Radicati Group in Palo Alto, Calif., spam is expected to account for 45% of the 10.9 trillion messages sent around the world in 2003.

One of the problems with spam, and the reason why there is so much of it, is that it is so easy to create. ­

You could easily become a spammer yourself. Let's say that you have a recipe from your grandmother for the best blueberry muffins ever created. A friend suggests that you sell the recipe for $5.

You decide that your friend might be on to something, so you send an e-mail to the 100 people in your personal e-mail address book with the subject line, "These Blueberry Muffins Have Been Described as Heaven -- You Can Have the Recipe for $5!" Your e-mail contains a link to your blueberry muffin Web site. As a result of your 100 e-mails, you get two orders and make $10.

As in, the Meat?
Where did the name "spam" come from? See Templetons.com to find out!

­ "Wow!" you think, "It cost me nothing to send those 100 e-mails, and I made $10. If I sent 1,000 e-mails I could make $100. If I sent a million e-mails I could make $100,000! I wonder where I could get a million e-mail addresses..."

So, how could you get 1 million e-mail addresses? Read on to find out.

A Plan for Spam

Like to build things? Try Hacker News.

August 2002

(This article describes the spam-filtering techniques used in the spamproof web-based mail reader we built to exercise Arc. An improved algorithm is described in Better Bayesian Filtering.)

I think it's possible to stop spam, and that content-based filters are the way to do it. The Achilles heel of the spammers is their message. They can circumvent any other barrier you set up. They have so far, at least. But they have to deliver their message, whatever it is. If we can write software that recognizes their messages, there is no way they can get around that.

_ _ _


To the recipient, spam is easily recognizable. If you hired someone to read your mail and discard the spam, they would have little trouble doing it. How much do we have to do, short of AI, to automate this process?

I think we will be able to solve the problem with fairly simple algorithms. In fact, I've found that you can filter present-day spam acceptably well using nothing more than a Bayesian combination of the spam probabilities of individual words. Using a slightly tweaked (as described below) Bayesian filter, we now miss less than 5 per 1000 spams, with 0 false positives.

The statistical approach is not usually the first one people try when they write spam filters. Most hackers' first instinct is to try to write software that recognizes individual properties of spam. You look at spams and you think, the gall of these guys to try sending me mail that begins "Dear Friend" or has a subject line that's all uppercase and ends in eight exclamation points. I can filter out that stuff with about one line of code.

And so you do, and in the beginning it works. A few simple rules will take a big bite out of your incoming spam. Merely looking for the word "click" will catch 79.7% of the emails in my spam corpus, with only 1.2% false positives.

I spent about six months writing software that looked for individual spam features before I tried the statistical approach. What I found was that recognizing that last few percent of spams got very hard, and that as I made the filters stricter I got more false positives.

False positives are innocent emails that get mistakenly identified as spams. For most users, missing legitimate email is an order of magnitude worse than receiving spam, so a filter that yields false positives is like an acne cure that carries a risk of death to the patient.

The more spam a user gets, the less likely he'll be to notice one innocent mail sitting in his spam folder. And strangely enough, the better your spam filters get, the more dangerous false positives become, because when the filters are really good, users will be more likely to ignore everything they catch.

I don't know why I avoided trying the statistical approach for so long. I think it was because I got addicted to trying to identify spam features myself, as if I were playing some kind of competitive game with the spammers. (Nonhackers don't often realize this, but most hackers are very competitive.) When I did try statistical analysis, I found immediately that it was much cleverer than I had been. It discovered, of course, that terms like "virtumundo" and "teens" were good indicators of spam. But it also discovered that "per" and "FL" and "ff0000" are good indicators of spam. In fact, "ff0000" (html for bright red) turns out to be as good an indicator of spam as any pornographic term.

_ _ _


Here's a sketch of how I do statistical filtering. I start with one corpus of spam and one of nonspam mail. At the moment each one has about 4000 messages in it. I scan the entire text, including headers and embedded html and javascript, of each message in each corpus. I currently consider alphanumeric characters, dashes, apostrophes, and dollar signs to be part of tokens, and everything else to be a token separator. (There is probably room for improvement here.) I ignore tokens that are all digits, and I also ignore html comments, not even considering them as token separators.

I count the number of times each token (ignoring case, currently) occurs in each corpus. At this stage I end up with two large hash tables, one for each corpus, mapping tokens to number of occurrences.

Next I create a third hash table, this time mapping each token to the probability that an email containing it is a spam, which I calculate as follows [1]: (let ((g (* 2 (or (gethash word good) 0))) (b (or (gethash word bad) 0))) (unless (&lt; (+ g b) 5) (max .01 (min .99 (float (/ (min 1 (/ b nbad)) (+ (min 1 (/ g ngood)) (min 1 (/ b nbad))))))))) where word is the token whose probability we're calculating, good and bad are the hash tables I created in the first step, and ngood and nbad are the number of nonspam and spam messages respectively.

I explained this as code to show a couple of important details. I want to bias the probabilities slightly to avoid false positives, and by trial and error I've found that a good way to do it is to double all the numbers in good. This helps to distinguish between words that occasionally do occur in legitimate email and words that almost never do. I only consider words that occur more than five times in total (actually, because of the doubling, occurring three times in nonspam mail would be enough). And then there is the question of what probability to assign to words that occur in one corpus but not the other. Again by trial and error I chose .01 and .99. There may be room for tuning here, but as the corpus grows such tuning will happen automatically anyway.

The especially observant will notice that while I consider each corpus to be a single long stream of text for purposes of counting occurrences, I use the number of emails in each, rather than their combined length, as the divisor in calculating spam probabilities. This adds another slight bias to protect against false positives.

When new mail arrives, it is scanned into tokens, and the most interesting fifteen tokens, where interesting is measured by how far their spam probability is from a neutral .5, are used to calculate the probability that the mail is spam. If probs is a list of the fifteen individual probabilities, you calculate the combined probability thus: (let ((prod (apply #'* probs))) (/ prod (+ prod (apply #'* (mapcar #'(lambda (x) (- 1 x)) probs))))) One question that arises in practice is what probability to assign to a word you've never seen, i.e. one that doesn't occur in the hash table of word probabilities. I've found, again by trial and error, that .4 is a good number to use. If you've never seen a word before, it is probably fairly innocent; spam words tend to be all too familiar.

There are examples of this algorithm being applied to actual emails in an appendix at the end.

I treat mail as spam if the algorithm above gives it a probability of more than .9 of being spam. But in practice it would not matter much where I put this threshold, because few probabilities end up in the middle of the range.

_ _ _


One great advantage of the statistical approach is that you don't have to read so many spams. Over the past six months, I've read literally thousands of spams, and it is really kind of demoralizing. Norbert Wiener said if you compete with slaves you become a slave, and there is something similarly degrading about competing with spammers. To recognize individual spam features you have to try to get into the mind of the spammer, and frankly I want to spend as little time inside the minds of spammers as possible.

But the real advantage of the Bayesian approach, of course, is that you know what you're measuring. Feature-recognizing filters like SpamAssassin assign a spam "score" to email. The Bayesian approach assigns an actual probability. The problem with a "score" is that no one knows what it means. The user doesn't know what it means, but worse still, neither does the developer of the filter. How many points should an email get for having the word "sex" in it? A probability can of course be mistaken, but there is little ambiguity about what it means, or how evidence should be combined to calculate it. Based on my corpus, "sex" indicates a .97 probability of the containing email being a spam, whereas "sexy" indicates .99 probability. And Bayes' Rule, equally unambiguous, says that an email containing both words would, in the (unlikely) absence of any other evidence, have a 99.97% chance of being a spam.

Because it is measuring probabilities, the Bayesian approach considers all the evidence in the email, both good and bad. Words that occur disproportionately rarely in spam (like "though" or "tonight" or "apparently") contribute as much to decreasing the probability as bad words like "unsubscribe" and "opt-in" do to increasing it. So an otherwise innocent email that happens to include the word "sex" is not going to get tagged as spam.

Ideally, of course, the probabilities should be calculated individually for each user. I get a lot of email containing the word "Lisp", and (so far) no spam that does. So a word like that is effectively a kind of password for sending mail to me. In my earlier spam-filtering software, the user could set up a list of such words and mail containing them would automatically get past the filters. On my list I put words like "Lisp" and also my zipcode, so that (otherwise rather spammy-sounding) receipts from online orders would get through. I thought I was being very clever, but I found that the Bayesian filter did the same thing for me, and moreover discovered of a lot of words I hadn't thought of.

When I said at the start that our filters let through less than 5 spams per 1000 with 0 false positives, I'm talking about filtering my mail based on a corpus of my mail. But these numbers are not misleading, because that is the approach I'm advocating: filter each user's mail based on the spam and nonspam mail he receives. Essentially, each user should have two delete buttons, ordinary delete and delete-as-spam. Anything deleted as spam goes into the spam corpus, and everything else goes into the nonspam corpus.

You could start users with a seed filter, but ultimately each user should have his own per-word probabilities based on the actual mail he receives. This (a) makes the filters more effective, (b) lets each user decide their own precise definition of spam, and (c) perhaps best of all makes it hard for spammers to tune mails to get through the filters. If a lot of the brain of the filter is in the individual databases, then merely tuning spams to get through the seed filters won't guarantee anything about how well they'll get through individual users' varying and much more trained filters.

Content-based spam filtering is often combined with a whitelist, a list of senders whose mail can be accepted with no filtering. One easy way to build such a whitelist is to keep a list of every address the user has ever sent mail to. If a mail reader has a delete-as-spam button then you could also add the from address of every email the user has deleted as ordinary trash.

I'm an advocate of whitelists, but more as a way to save computation than as a way to improve filtering. I used to think that whitelists would make filtering easier, because you'd only have to filter email from people you'd never heard from, and someone sending you mail for the first time is constrained by convention in what they can say to you. Someone you already know might send you an email talking about sex, but someone sending you mail for the first time would not be likely to. The problem is, people can have more than one email address, so a new from-address doesn't guarantee that the sender is writing to you for the first time. It is not unusual for an old friend (especially if he is a hacker) to suddenly send you an email with a new from-address, so you can't risk false positives by filtering mail from unknown addresses especially stringently.

In a sense, though, my filters do themselves embody a kind of whitelist (and blacklist) because they are based on entire messages, including the headers. So to that extent they "know" the email addresses of trusted senders and even the routes by which mail gets from them to me. And they know the same about spam, including the server names, mailer versions, and protocols.

_ _ _


If I thought that I could keep up current rates of spam filtering, I would consider this problem solved. But it doesn't mean much to be able to filter out most present-day spam, because spam evolves. Indeed, most antispam techniques so far have been like pesticides that do nothing more than create a new, resistant strain of bugs.

I'm more hopeful about Bayesian filters, because they evolve with the spam. So as spammers start using "c0ck" instead of "cock" to evade simple-minded spam filters based on individual words, Bayesian filters automatically notice. Indeed, "c0ck" is far more damning evidence than "cock", and Bayesian filters know precisely how much more.

Still, anyone who proposes a plan for spam filtering has to be able to answer the question: if the spammers knew exactly what you were doing, how well could they get past you? For example, I think that if checksum-based spam filtering becomes a serious obstacle, the spammers will just switch to mad-lib techniques for generating message bodies.

To beat Bayesian filters, it would not be enough for spammers to make their emails unique or to stop using individual naughty words. They'd have to make their mails indistinguishable from your ordinary mail. And this I think would severely constrain them. Spam is mostly sales pitches, so unless your regular mail is all sales pitches, spams will inevitably have a different character. And the spammers would also, of course, have to change (and keep changing) their whole infrastructure, because otherwise the headers would look as bad to the Bayesian filters as ever, no matter what they did to the message body. I don't know enough about the infrastructure that spammers use to know how hard it would be to make the headers look innocent, but my guess is that it would be even harder than making the message look innocent.

Assuming they could solve the problem of the headers, the spam of the future will probably look something like this: Hey there. Thought you should check out the following: http://www.27meg.com/foo because that is about as much sales pitch as content-based filtering will leave the spammer room to make. (Indeed, it will be hard even to get this past filters, because if everything else in the email is neutral, the spam probability will hinge on the url, and it will take some effort to make that look neutral.)

Spammers range from businesses running so-called opt-in lists who don't even try to conceal their identities, to guys who hijack mail servers to send out spams promoting porn sites. If we use filtering to whittle their options down to mails like the one above, that should pretty much put the spammers on the "legitimate" end of the spectrum out of business; they feel obliged by various state laws to include boilerplate about why their spam is not spam, and how to cancel your "subscription," and that kind of text is easy to recognize.

(I used to think it was naive to believe that stricter laws would decrease spam. Now I think that while stricter laws may not decrease the amount of spam that spammers send, they can certainly help filters to decrease the amount of spam that recipients actually see.)

All along the spectrum, if you restrict the sales pitches spammers can make, you will inevitably tend to put them out of business. That word business is an important one to remember. The spammers are businessmen. They send spam because it works. It works because although the response rate is abominably low (at best 15 per million, vs 3000 per million for a catalog mailing), the cost, to them, is practically nothing. The cost is enormous for the recipients, about 5 man-weeks for each million recipients who spend a second to delete the spam, but the spammer doesn't have to pay that.

Sending spam does cost the spammer something, though. [2] So the lower we can get the response rate-- whether by filtering, or by using filters to force spammers to dilute their pitches-- the fewer businesses will find it worth their while to send spam.

The reason the spammers use the kinds of sales pitches that they do is to increase response rates. This is possibly even more disgusting than getting inside the mind of a spammer, but let's take a quick look inside the mind of someone who responds to a spam. This person is either astonishingly credulous or deeply in denial about their sexual interests. In either case, repulsive or idiotic as the spam seems to us, it is exciting to them. The spammers wouldn't say these things if they didn't sound exciting. And "thought you should check out the following" is just not going to have nearly the pull with the spam recipient as the kinds of things that spammers say now. Result: if it can't contain exciting sales pitches, spam becomes less effective as a marketing vehicle, and fewer businesses want to use it.

That is the big win in the end. I started writing spam filtering software because I didn't want have to look at the stuff anymore. But if we get good enough at filtering out spam, it will stop working, and the spammers will actually stop sending it.

_ _ _


Of all the approaches to fighting spam, from software to laws, I believe Bayesian filtering will be the single most effective. But I also think that the more different kinds of antispam efforts we undertake, the better, because any measure that constrains spammers will tend to make filtering easier. And even within the world of content-based filtering, I think it will be a good thing if there are many different kinds of software being used simultaneously. The more different filters there are, the harder it will be for spammers to tune spams to get through them.



Appendix: Examples of Filtering

Here is an example of a spam that arrived while I was writing this article. The fifteen most interesting words in this spam are: qvp0045 indira mx-05 intimail $7500 freeyankeedom cdo bluefoxmedia jpg unsecured platinum 3d0 qves 7c5 7c266675 The words are a mix of stuff from the headers and from the message body, which is typical of spam. Also typical of spam is that every one of these words has a spam probability, in my database, of .99. In fact there are more than fifteen words with probabilities of .99, and these are just the first fifteen seen.

Unfortunately that makes this email a boring example of the use of Bayes' Rule. To see an interesting variety of probabilities we have to look at this actually quite atypical spam.

The fifteen most interesting words in this spam, with their probabilities, are: madam 0.99 promotion 0.99 republic 0.99 shortest 0.047225013 mandatory 0.047225013 standardization 0.07347802 sorry 0.08221981 supported 0.09019077 people's 0.09019077 enter 0.9075001 quality 0.8921298 organization 0.12454646 investment 0.8568143 very 0.14758544 valuable 0.82347786 This time the evidence is a mix of good and bad. A word like "shortest" is almost as much evidence for innocence as a word like "madam" or "promotion" is for guilt. But still the case for guilt is stronger. If you combine these numbers according to Bayes' Rule, the resulting probability is .9027.

"Madam" is obviously from spams beginning "Dear Sir or Madam." They're not very common, but the word "madam" never occurs in my legitimate email, and it's all about the ratio.

"Republic" scores high because it often shows up in Nigerian scam emails, and also occurs once or twice in spams referring to Korea and South Africa. You might say that it's an accident that it thus helps identify this spam. But I've found when examining spam probabilities that there are a lot of these accidents, and they have an uncanny tendency to push things in the right direction rather than the wrong one. In this case, it is not entirely a coincidence that the word "Republic" occurs in Nigerian scam emails and this spam. There is a whole class of dubious business propositions involving less developed countries, and these in turn are more likely to have names that specify explicitly (because they aren't) that they are republics.[3]

On the other hand, "enter" is a genuine miss. It occurs mostly in unsubscribe instructions, but here is used in a completely innocent way. Fortunately the statistical approach is fairly robust, and can tolerate quite a lot of misses before the results start to be thrown off.

For comparison, here is an example of that rare bird, a spam that gets through the filters. Why? Because by sheer chance it happens to be loaded with words that occur in my actual email: perl 0.01 python 0.01 tcl 0.01 scripting 0.01 morris 0.01 graham 0.01491078 guarantee 0.9762507 cgi 0.9734398 paul 0.027040077 quite 0.030676773 pop3 0.042199217 various 0.06080265 prices 0.9359873 managed 0.06451222 difficult 0.071706355 There are a couple pieces of good news here. First, this mail probably wouldn't get through the filters of someone who didn't happen to specialize in programming languages and have a good friend called Morris. For the average user, all the top five words here would be neutral and would not contribute to the spam probability.

Second, I think filtering based on word pairs (see below) might well catch this one: "cost effective", "setup fee", "money back" -- pretty incriminating stuff. And of course if they continued to spam me (or a network I was part of), "Hostex" itself would be recognized as a spam term.

Finally, here is an innocent email. Its fifteen most interesting words are as follows: continuation 0.01 describe 0.01 continuations 0.01 example 0.033600237 programming 0.05214485 i'm 0.055427782 examples 0.07972858 color 0.9189189 localhost 0.09883721 hi 0.116539136 california 0.84421706 same 0.15981844 spot 0.1654587 us-ascii 0.16804294 what 0.19212411 Most of the words here indicate the mail is an innocent one. There are two bad smelling words, "color" (spammers love colored fonts) and "California" (which occurs in testimonials and also in menus in forms), but they are not enough to outweigh obviously innocent words like "continuation" and "example".

It's interesting that "describe" rates as so thoroughly innocent. It hasn't occurred in a single one of my 4000 spams. The data turns out to be full of such surprises. One of the things you learn when you analyze spam texts is how narrow a subset of the language spammers operate in. It's that fact, together with the equally characteristic vocabulary of any individual user's mail, that makes Bayesian filtering a good bet.

Appendix: More Ideas

One idea that I haven't tried yet is to filter based on word pairs, or even triples, rather than individual words. This should yield a much sharper estimate of the probability. For example, in my current database, the word "offers" has a probability of .96. If you based the probabilities on word pairs, you'd end up with "special offers" and "valuable offers" having probabilities of .99 and, say, "approach offers" (as in "this approach offers") having a probability of .1 or less.

The reason I haven't done this is that filtering based on individual words already works so well. But it does mean that there is room to tighten the filters if spam gets harder to detect. (Curiously, a filter based on word pairs would be in effect a Markov-chaining text generator running in reverse.)

Specific spam features (e.g. not seeing the recipient's address in the to: field) do of course have value in recognizing spam. They can be considered in this algorithm by treating them as virtual words. I'll probably do this in future versions, at least for a handful of the most egregious spam indicators. Feature-recognizing spam filters are right in many details; what they lack is an overall discipline for combining evidence.

Recognizing nonspam features may be more important than recognizing spam features. False positives are such a worry that they demand extraordinary measures. I will probably in future versions add a second level of testing designed specifically to avoid false positives. If a mail triggers this second level of filters it will be accepted even if its spam probability is above the threshold.

I don't expect this second level of filtering to be Bayesian. It will inevitably be not only ad hoc, but based on guesses, because the number of false positives will not tend to be large enough to notice patterns. (It is just as well, anyway, if a backup system doesn't rely on the same technology as the primary system.)

Another thing I may try in the future is to focus extra attention on specific parts of the email. For example, about 95% of current spam includes the url of a site they want you to visit. (The remaining 5% want you to call a phone number, reply by email or to a US mail address, or in a few cases to buy a certain stock.) The url is in such cases practically enough by itself to determine whether the email is spam.

Domain names differ from the rest of the text in a (non-German) email in that they often consist of several words stuck together. Though computationally expensive in the general case, it might be worth trying to decompose them. If a filter has never seen the token "xxxporn" before it will have an individual spam probability of .4, whereas "xxx" and "porn" individually have probabilities (in my corpus) of .9889 and .99 respectively, and a combined probability of .9998.

I expect decomposing domain names to become more important as spammers are gradually forced to stop using incriminating words in the text of their messages. (A url with an ip address is of course an extremely incriminating sign, except in the mail of a few sysadmins.)

It might be a good idea to have a cooperatively maintained list of urls promoted by spammers. We'd need a trust metric of the type studied by Raph Levien to prevent malicious or incompetent submissions, but if we had such a thing it would provide a boost to any filtering software. It would also be a convenient basis for boycotts.

Another way to test dubious urls would be to send out a crawler to look at the site before the user looked at the email mentioning it. You could use a Bayesian filter to rate the site just as you would an email, and whatever was found on the site could be included in calculating the probability of the email being a spam. A url that led to a redirect would of course be especially suspicious.

One cooperative project that I think really would be a good idea would be to accumulate a giant corpus of spam. A large, clean corpus is the key to making Bayesian filtering work well. Bayesian filters could actually use the corpus as input. But such a corpus would be useful for other kinds of filters too, because it could be used to test them.

Creating such a corpus poses some technical problems. We'd need trust metrics to prevent malicious or incompetent submissions, of course. We'd also need ways of erasing personal information (not just to-addresses and ccs, but also e.g. the arguments to unsubscribe urls, which often encode the to-address) from mails in the corpus. If anyone wants to take on this project, it would be a good thing for the world.

Appendix: Defining Spam

I think there is a rough consensus on what spam is, but it would be useful to have an explicit definition. We'll need to do this if we want to establish a central corpus of spam, or even to compare spam filtering rates meaningfully.

To start with, spam is not unsolicited commercial email. If someone in my neighborhood heard that I was looking for an old Raleigh three-speed in good condition, and sent me an email offering to sell me one, I'd be delighted, and yet this email would be both commercial and unsolicited. The defining feature of spam (in fact, its raison d'etre) is not that it is unsolicited, but that it is automated.

It is merely incidental, too, that spam is usually commercial. If someone started sending mass email to support some political cause, for example, it would be just as much spam as email promoting a porn site.

I propose we define spam as unsolicited automated email. This definition thus includes some email that many legal definitions of spam don't. Legal definitions of spam, influenced presumably by lobbyists, tend to exclude mail sent by companies that have an "existing relationship" with the recipient. But buying something from a company, for example, does not imply that you have solicited ongoing email from them. If I order something from an online store, and they then send me a stream of spam, it's still spam.

Companies sending spam often give you a way to "unsubscribe," or ask you to go to their site and change your "account preferences" if you want to stop getting spam. This is not enough to stop the mail from being spam. Not opting out is not the same as opting in. Unless the recipient explicitly checked a clearly labelled box (whose default was no) asking to receive the email, then it is spam.

In some business relationships, you do implicitly solicit certain kinds of mail. When you order online, I think you implicitly solicit a receipt, and notification when the order ships. I don't mind when Verisign sends me mail warning that a domain name is about to expire (at least, if they are the actual registrar for it). But when Verisign sends me email offering a FREE Guide to Building My E-Commerce Web Site, that's spam.

Notes:

[1] The examples in this article are translated into Common Lisp for, believe it or not, greater accessibility. The application described here is one that we wrote in order to test a new Lisp dialect called Arc that is not yet released.

[2] Currently the lowest rate seems to be about $200 to send a million spams. That's very cheap, 1/50th of a cent per spam. But filtering out 95% of spam, for example, would increase the spammers' cost to reach a given audience by a factor of 20. Few can have margins big enough to absorb that.

[3] As a rule of thumb, the more qualifiers there are before the name of a country, the more corrupt the rulers. A country called The Socialist People's Democratic Republic of X is probably the last place in the world you'd want to live.

Thanks to Sarah Harlin for reading drafts of this; Daniel Giffin (who is also writing the production Arc interpreter) for several good ideas about filtering and for creating our mail infrastructure; Robert Morris, Trevor Blackwell and Erann Gat for many discussions about spam; Raph Levien for advice about trust metrics; and Chip Coldwell and Sam Steingold for advice about statistics.

Note

This is the home page for the open-source Apache SpamAssassin Project. There are also numerous prepackaged versions for Windows, commercial versions, and specialized front-ends.

If you were sent here because you received an e-mail message which was modified by SpamAssassin, please read this page.

Latest News

2011-06-16: SpamAssassin 3.3.2 has been released, a minor new release primarily to support perl-5.12 and later. Visit the downloads page to pick it up, and for more info.

(Older news items can be read at the News and Announcements page. Atom feed )

Features

* Wide-spectrum: SpamAssassin uses a wide variety of local and network tests to identify spam signatures. This makes it harder for spammers to identify one aspect which they can craft their messages to work around.
* Free software: it is distributed under the same terms and conditions as other popular open-source software packages such as the Apache web server.
* Easy to extend: Anti-spam tests and configuration are stored in plain text, making it easy to configure and add new rules.
* Flexible: SpamAssassin encapsulates its logic in a well-designed, abstract API so it can be integrated anywhere in the email stream. The Mail::SpamAssassin classes can be used on a wide variety of email systems including procmail, sendmail, Postfix, qmail, and many others.
* Easy Configuration: SpamAssassin requires very little configuration; you do not need to continually update it with details of your mail accounts, mailing list memberships, etc. Once classified, site and user-specific policies can then be applied against spam. Policies can be applied on both mail servers and later using the user's own mail user-agent application.

How Spammers Get Addresses

As it turns out, there are hundreds of companies that will sell you CDs filled with millions of valid e-mail addresses. With Microsoft Word you could easily format those addresses into lines of 100 addresses each, and then cut and paste those lines into the "To:" field of any normal e-mail program. Every time you push the "Send" button, which would be about once every 5 seconds, you would make $10. You would be making something like $700 per hour.

This is the problem with spam. It is incredibly easy for you to send it. It costs you practically nothing to send it. And even with a response rate as low as one sale out of 10,000 e-mails, it can be quite lucrative for you to send it. Therefore, if you don't mind the fact that you are creating e-mail pollution for millions of people, you might decide to send e-mail messages about your grandmother's muffins all day long.

Where does a company get millions of valid e-mail addresses to put on a CD and sell to you? There are a number of primary sources.

The first is newsgroups and chat rooms, especially on big sites like AOL. People (especially first-time users) often use their screen names, or leave their actual e-mail addresses, in newsgroups. Spammers use pieces of software to extract the screen names and e-mail addresses automatically.

The second source for e-mail addresses is the Web itself. There are tens of millions of Web sites, and spammers can create search engines that spider the Web specifically looking for the telltale "@" sign that indicates an e-mail address. The programs that do the spidering are often called spambots.

The third source is sites created specifically to attract e-mail addresses. For example, a spammer creates a site that says, "Win $1 million!! ! Just type your e-mail address here!" In the past, lots of large sites also sold the e-mail addresses of their members. Or the sites created "opt-in" e-mail lists by asking, "Would you like to receive e-mail newsletters from our partners?" If you answered yes, your address was then sold to a spammer.

Probably the most common source of e-mail addresses, however, is a "dictionary" search of the e-mail servers of large e-mail hosting companies like MSN, AOL or Hotmail. In the article Hotmail: A Spammer's Paradise?, the author describes the process:

A dictionary attack utilizes software that opens a connection to the target mail server and then rapidly submits millions of random e-mail addresses. Many of these addresses have slight variations, such as "[email protected]" and "[email protected]." The software then records which addresses are "live," and adds those addresses to the spammer's list. These lists are typically resold to many other spammers.

E-mail addresses generally are not private (just like your phone number is not private if it is listed in the phone book). Once a spammer gets a hold of your e-mail address and starts sharing it with other spammers, you are likely to get a lot of spam.

CAN-SPAM Act: A Compliance Guide for Business [PDF]

Do you use email in your business? The CAN-SPAM Act, a law that sets the rules for commercial email, establishes requirements for commercial messages, gives recipients the right to have you stop emailing them, and spells out tough penalties for violations.

Despite its name, the CAN-SPAM Act doesn’t apply just to bulk email. It covers all commercial messages, which the law defines as “any electronic mail message the primary purpose of which is the commercial advertisement or promotion of a commercial product or service,” including email that promotes content on commercial websites. The law makes no exception for business-to-business email. That means all email – for example, a message to former customers announcing a new product line – must comply with the law.

Each separate email in violation of the CAN-SPAM Act is subject to penalties of up to $16,000, so non-compliance can be costly. But following the law isn’t complicated. Here’s a rundown of CAN-SPAM’s main requirements:

1. Don’t use false or misleading header information. Your “From,” “To,” “Reply-To,” and routing information – including the originating domain name and email address – must be accurate and identify the person or business who initiated the message.
2. Don’t use deceptive subject lines. The subject line must accurately reflect the content of the message.
3. Identify the message as an ad. The law gives you a lot of leeway in how to do this, but you must disclose clearly and conspicuously that your message is an advertisement.
4. Tell recipients where you’re located. Your message must include your valid physical postal address. This can be your current street address, a post office box you’ve registered with the U.S. Postal Service, or a private mailbox you’ve registered with a commercial mail receiving agency established under Postal Service regulations.
5. Tell recipients how to opt out of receiving future email from you. Your message must include a clear and conspicuous explanation of how the recipient can opt out of getting email from you in the future. Craft the notice in a way that’s easy for an ordinary person to recognize, read, and understand. Creative use of type size, color, and location can improve clarity. Give a return email address or another easy Internet-based way to allow people to communicate their choice to you. You may create a menu to allow a recipient to opt out of certain types of messages, but you must include the option to stop all commercial messages from you. Make sure your spam filter doesn’t block these opt-out requests.
6. Honor opt-out requests promptly. Any opt-out mechanism you offer must be able to process opt-out requests for at least 30 days after you send your message. You must honor a recipient’s opt-out request within 10 business days. You can’t charge a fee, require the recipient to give you any personally identifying information beyond an email address, or make the recipient take any step other than sending a reply email or visiting a single page on an Internet website as a condition for honoring an opt-out request. Once people have told you they don’t want to receive more messages from you, you can’t sell or transfer their email addresses, even in the form of a mailing list. The only exception is that you may transfer the addresses to a company you’ve hired to help you comply with the CAN-SPAM Act.
7. Monitor what others are doing on your behalf. The law makes clear that even if you hire another company to handle your email marketing, you can’t contract away your legal responsibility to comply with the law. Both the company whose product is promoted in the message and the company that actually sends the message may be held legally responsible.

Need more information?

Here are the answers to some questions businesses have had about complying with the CAN-SPAM Act.
Q. How do I know if the CAN-SPAM Act covers email my business is sending?

A. What matters is the “primary purpose” of the message. To determine the primary purpose, remember that an email can contain three different types of information:

* Commercial content – which advertises or promotes a commercial product or service, including content on a website operated for a commercial purpose;
* Transactional or relationship content – which facilitates an already agreed-upon transaction or updates a customer about an ongoing transaction; and
* Other content – which is neither commercial nor transactional or relationship.

If the message contains only commercial content, its primary purpose is commercial and it must comply with the requirements of CAM-SPAM. If it contains only transactional or relationship content, its primary purpose is transactional or relationship. In that case, it may not contain false or misleading routing information, but is otherwise exempt from most provisions of the CAN-SPAM Act.

Q. How do I know if what I’m sending is a transactional or relationship message?

A. The primary purpose of an email is transactional or relationship if it consists only of content that:

1. facilitates or confirms a commercial transaction that the recipient already has agreed to;
2. gives warranty, recall, safety, or security information about a product or service;
3. gives information about a change in terms or features or account balance information regarding a membership, subscription, account, loan or other ongoing commercial relationship;
4. provides information about an employment relationship or employee benefits; or
5. delivers goods or services as part of a transaction that the recipient already has agreed to.

Q. What if the message combines commercial content and transactional or relationship content?

A. It’s common for email sent by businesses to mix commercial content and transactional or relationship content. When an email contains both kinds of content, the primary purpose of the message is the deciding factor. Here’s how to make that determination: If a recipient reasonably interpreting the subject line would likely conclude that the message contains an advertisement or promotion for a commercial product or service or if the message’s transactional or relationship content does not appear mainly at the beginning of the message, the primary purpose of the message is commercial. So, when a message contains both kinds of content – commercial and transactional or relationship – if the subject line would lead the recipient to think it’s a commercial message, it’s a commercial message for CAN-SPAM purposes. Similarly, if the bulk of the transactional or relationship part of the message doesn’t appear at the beginning, it’s a commercial message under the CAN-SPAM Act.

Here's an example:
MESSAGE A:

TO: Jane Smith

FR: XYZ Distributing

RE: Your Account Statement

We shipped your order of 25,000 deluxe widgets to your Springfield warehouse on June 1st. We hope you received them in good working order. Please call our Customer Service Office at (877) 555-7726 if any widgets were damaged in transit. Per our contract, we must receive your payment of $1,000 by June 30th. If not, we will impose a 10% surcharge for late payment. If you have any questions, please contact our Accounts Receivable Department.

Visit our website for our exciting new line of mini-widgets!

MESSAGE A is most likely a transactional or relationship message subject only to CAN-SPAM’s requirement of truthful routing information. One important factor is that information about the customer’s account is at the beginning of the message and the brief commercial portion of the message is at the end.
MESSAGE B:

TO: Jane Smith

FR: XYZ Distributing

RE: Your Account Statement

We offer a wide variety of widgets in the most popular designer colors and styles – all at low, low discount prices. Visit our website for our exciting new line of mini-widgets!

Sizzling Summer Special: Order by June 30th and all waterproof commercial-grade super-widgets are 20% off. Show us a bid from one of our competitors and we’ll match it. XYZ Distributing will not be undersold.

Your order has been filled and will be delivered on Friday, June 1st.

MESSAGE MESSAGE B is most likely a commercial message subject to all CAN-SPAM's requirements. Although the subject line is “Your Account Statement” – generally a sign of a transactional or relationship message – the information at the beginning of the message is commercial in nature and the brief transactional or relationship portion of the message is at the end.

Q. What if the message combines elements of both a commercial message and a message with content defined as "other"?

A. In that case, the primary purpose of the message is commercial and the provisions of the CAN-SPAM Act apply if:

* A recipient reasonably interpreting the subject line would likely conclude that the message advertises or promotes a commercial product or service; and
* A recipient reasonably interpreting the body of the message would likely conclude that the primary purpose of the message is to advertise or promote a product or service.

Factors relevant to that interpretation include the location of the commercial content (for example, is it at the beginning of the message?); how much of the message is dedicated to commercial content; and how color, graphics, type size, style, etc., are used to highlight the commercial content.

Q. What if the email includes information from more than one company? Who is the “sender” responsible for CAN-SPAM compliance?

A. If an email advertises or promotes the goods, services, or websites of more than one marketer, there’s a straightforward method for determining who’s responsible for the duties the CAN-SPAM Act imposes on “senders” of commercial email. Marketers whose goods, services, or websites are advertised or promoted in a message can designate one of the marketers as the “sender” for purposes of CAN-SPAM compliance as long as the designated sender:

* meets the CAN-SPAM Act’s definition of “sender,” meaning that they initiate a commercial message advertising or promoting their own goods, services, or website;
* is specifically identified in the “from” line of the message; and
* complies with the “initiator” provisions of the Act – for example, making sure the email does not contain deceptive transmission information or a deceptive subject heading, and ensuring that the email includes a valid postal address, a working opt-out link, and proper identification of the message’s commercial or sexually explicit nature.

If the designated sender doesn’t comply with the responsibilities the law gives to initiators, all marketers in the message may be held liable as senders.

Q. My company sends email with a link so that recipients can forward the message to others. Who is responsible for CAN-SPAM compliance for these “Forward to a Friend” messages?

A. Whether a seller or forwarder is a “sender” or “initiator” depends on the facts. So deciding if the CAN-SPAM Act applies to a commercial “forward-to-a-friend” message often depends on whether the seller has offered to pay the forwarder or give the forwarder some other benefit. For example, if the seller offers money, coupons, discounts, awards, additional entries in a sweepstakes, or the like in exchange for forwarding a message, the seller may be responsible for compliance. Or if a seller pays or give a benefit to someone in exchange for generating traffic to a website or for any form of referral, the seller is likely to have compliance obligations under the CAN-SPAM Act.

Q. What are the penalties for violating the CAN-SPAM Act?

A. Each separate email in violation of the law is subject to penalties of up to $16,000, and more than one person may be held responsible for violations. For example, both the company whose product is promoted in the message and the company that originated the message may be legally responsible. Email that makes misleading claims about products or services also may be subject to laws outlawing deceptive advertising, like Section 5 of the FTC Act. The CAN-SPAM Act has certain aggravated violations that may give rise to additional fines. The law provides for criminal penalties – including imprisonment – for:

* accessing someone else’s computer to send spam without permission,
* using false information to register for multiple email accounts or domain names,
* relaying or retransmitting multiple spam messages through a computer to mislead others about the origin of the message,
* harvesting email addresses or generating them through a dictionary attack (the practice of sending email to addresses made up of random letters and numbers in the hope of reaching valid ones), and
* taking advantage of open relays or open proxies without permission.

Q. Are there separate rules that apply to sexually explicit email?

A. Yes, and the FTC has issued a rule under the CAN-SPAM Act that governs these messages. Messages with sexually oriented material must include the warning “SEXUALLY-EXPLICIT:” at the beginning of the subject line. In addition, the rule requires the electronic equivalent of a “brown paper wrapper” in the body of the message. When a recipient opens the message, the only things that may be viewable on the recipient’s screen are:

1. the words “SEXUALLY-EXPLICIT:”; and
2. the same information required in any other commercial email: a disclosure that the message is an ad, the sender’s physical postal address, and the procedure for how recipients can opt out of receiving messages from this sender in the future.

No graphics are allowed on the “brown paper wrapper.” This provision makes sure that recipients cannot view sexually explicit content without an affirmative act on their part – for example, scrolling down or clicking on a link. However, this requirement does not apply if the person receiving the message has already given affirmative consent to receive the sender’s sexually oriented messages.

Q. How can I comment about the effect of the CAN-SPAM Act on my business?

A. The National Small Business Ombudsman collects comments from small businesses about federal compliance and enforcement activities. To comment, call 1-888-REG-FAIR (1-888-734-3247) or visit www.sba.gov/ombudsman.

For More Information

The FTC works for the consumer to prevent fraudulent, deceptive, and unfair business practices in the marketplace and to provide information to help consumers spot, stop, and avoid them. To file a complaint or to get free information on consumer issues, visit ftc.gov or call toll-free, 1-877-FTC-HELP (1-877-382-4357); TTY: 1-866-653-4261. The FTC enters consumer complaints into the Consumer Sentinel Network, a secure online database and investigative tool used by hundreds of civil and criminal law enforcement agencies in the U.S. and abroad.
Your Opportunity to Comment

The National Small Business Ombudsman and 10 Regional Fairness Boards collect comments from small businesses about federal compliance and enforcement activities. Each year, the Ombudsman evaluates the conduct of these activities and rates each agency's responsiveness to small businesses. Small businesses can comment to the Ombudsman without fear of reprisal. To comment, call toll-free 1-888-REGFAIR (1-888-734-3247) or go to www.sba.gov/ombudsman.



September 2009

The Big Spamming Companies
If you would like to send a lot of spam, then there are a number of companies set up to send "bulk e-mail." The largest of these companies are able to send billions of spam e-mail messages a day. They increasingly operate out of foreign countries to avoid U.S. laws and lawsuits trying to block spam. Detroit Free Press: Spam king lives large off others' e-mail troubles describes a typical spam operation:

The computers in Ralsky's basement control 190 e-mail servers -- 110 located in Southfield, 50 in Dallas and 30 more in Canada, China, Russia and India. Each computer, he said, is capable of sending out 650,000 messages every hour -- more than a billion a day -- routed through overseas Internet companies Ralsky said are eager to sell him bandwidth.

There are hundreds of companies like this. For example, here's a paid ad that appeared on Google in August 2003:

bulk e-mail ad

The company is offering to send 500,000 e-mails for $99 and says, "Imagine emailing 500,000 recipients and 1 out of every 1,000 orders your product, that's 500 new orders!" Similarly, if you type "bulk e-mail" as a search term in Google, you get this assortment of paid ads in late August 2003:

bulk e-mail ads

All of these vendors are claiming that they are "spam-free." That is, they claim that they use e-mail lists where the recipients have specifically requested to receive bulk e-mail. This is often referred to "opt-in" e-mail. You may have ordered a product or filled out an online form that had a checkbox at the bottom that said, "Please unclick this check box if you would not like to receive e-mail from our partners," or something to that effect. You either did not see that checkbox because it was way at the bottom of the form, or you misread it. If your name gets onto the wrong opt-in lists, then you will receive a great deal of spam.

It's important to note that checking a checkbox like this doesn't always lead to spam. In some cases, non-spamming sites are only asking you to opt in for good, non-advertorial e-mail, such as a newsletter. These sites may use pre-checked boxes for convenience.

Now, let's find out what you can do to stop spam.

Come See What's Inside!

The SPAM® Museum is dedicated to the delicious meat first created at the Hormel Foods Corporation plant in 1937. It’s a fun and informational destination with interactive and educational elements designed for all ages.

How to Stop Spam

spam agent spam filtering software
Spam filtering software is one way to decrease spam.
The best technology that is currently available to stop spam is spam filtering software. The simplest filters use keywords such as "sex,", "xxx," "viagra," etc., in the subject line to attempt to identify and delete spam. These simple filters are easy to sidestep by spelling "sex" as "s-e-x." There are, of course, thousands of ways to spell "sex" if you are willing to add extra characters like that, and it is difficult for the simple filters to keep up. Also, simple filters are most likely to block "real" e-mail that you do want to receive. For example, if your friend sends you her favorite recipe for baked chicken breasts, the filter blocks the e-mail because of the word "breasts."

More advanced filters, known as heuristic filters and Bayesian filters, try to take this simple approach quite a bit further to statistically identify spam based on word patterns or word frequency. But there are still ways to get around them (mainly by using short messages).

Large ISPs tried blocking multiple e-mails with the same subject line or message body. This had the unwanted side-effect of blocking e-mail newsletters, so ISPs made "white lists" to identify legitimate newsletter senders. Then spammers sidestepped the issue by inserting different random characters into each subject line and message body. That's why you get e-mail messages with subject lines like:

Women Wanted puklq

The word "puklq" is random, and it is different on every e-mail the spammer sends.

There are several organizations that publish lists of IP addresses that are used by spammers. Any large spammer will have an array of server machines blasting out spam messages, and each server machine has its own IP address. Once spam is detected from an IP address, that IP address is put in a list (Spamhaus.org is one of many organizations that maintain such lists). Companies that host e-mail accounts can look at the sending IP address of every e-mail and filter out those that appear in the Spamhaus.org list.

Spammers get around this approach in two different ways. First, they change their IP addresses frequently. The unfortunate problem with this approach is that the old IP addresses that spammers discard get recycled, and the people who get these discarded IP addresses find them to be useless -- they are tainted by their former association with spam, and cannot be used for sending legitimate e-mail.

Lately, spammers have started to get more aggressive. For example, it is thought that recent viruses like SoBig.F were sent out specifically to recruit "zombie machines" for spammers. The zombie machines are generally personal computers owned by unsuspecting private citizens who happened to contract the SoBig virus. The virus opens their machine up to spammers, who can then route spam e-mails through their machines. Since the IP addresses of these machines are new, they do not appear in the IP address blacklists and millions of spam e-mails can route through them before they get blacklisted. The zombie machines have also been uses for denial of service attacks on places like Spamhaus.org.

Another front in the war against spam is legislation. For example, it has been suggested that the U.S. federal government set up a national "do not spam" list identical to the national Do Not Call list designed to block telemarketers. However, it is believed by most people that spammers are so obnoxious that they would set up spam servers in foreign countries and actually use the "do not spam" list as a source of fresh e-mail addresses.

Another solution would be an "opt-in" list. Under this proposal, only those people who specifically request spam e-mail would get it. However, the United States congress seems to be heading in the opposite direction. As noted on Spamhaus.org:

With all of Europe set to implement Opt-in legislation by October, Europe has taken the lead in banning spam. But the United States is going in the opposite direction, legislating Opt-out instead of Opt-in and looks set to explode the spam problem many times worse than it is today, incredibly by actually legalizing spam instead of banning it. US Congress is just months away from giving Unsolicited Bulk e-mail the green light and unleashing the spamming power of 23 Million American businesses onto an Internet which already can not cope with the billions of unsolicited bulk mailings sent by just 200 businesses. As spammers applaud the introduction of pro-spam Bills, we look at why spammers now cheer so loudly for Congressman Billy Tauzin.

The final front in the war on spam is the elimination of e-mail in the traditional sense. Many businesses are being forced to take this approach. Even the White House has been forced to follow this path. Today, if you want to send e-mail to the president of the United States, you do it by filling out an online form. Even HowStuffWorks has been forced to use forms. It used to be that you could send e-mail directly to individual HowStuffWorks staff and departments, but those e-mail addresses started to receive so much spam that we now use a set of online forms, instead.

That may be what happens to all e-mail in the long run. The amount of spam, and the inability to control that spam, may become so unmanageable that the traditional e-mail system we know today collapses and gets replaced either with forms or with a set of advanced, secure servers that put spammers out of business.

For more information on spam and related topics, check out the links on the next page.

*the more you know apparently*

This thread is mine for the taking tomorrow.

safety Safety
by spam
Haven't checked oAuth approvals lately? Here's more info on how to check this out: bit.ly/capEf3

safety Safety
by spam
Friendly reminder that Twitter doesn't send fake emails about unread messages - check bit.ly/asnPFd to learn more.

safety Safety
by spam
Received a fake email about warnings from Twitter? Forward it to [email protected] and learn more here: bit.ly/asnPFd

safety Safety
by spam
Please see status.twitter.com/post/119287388… for info on the latest (resolved) attack on Twitter.