DomainKeys for Postfix

If you're reading this you probably know that spammers give all websites a bad name. There are various ways to try and combat this such as spf, reverse dns, and DomainKeys. We'll be focusing on the latter and it's implementation on Postfix 2.3.

DomainKeys uses asymmetrical (public key/private key) encryption to verify the sender of the email is valid. While it is an open standard, as far as we can tell, the only large company to adopt this is it's creator Yahoo!. Expect this to improve delivery rates to only Yahoo! Mail customers.

We will be implementing domain keys as basically a mail filter. Outbound mail will be signed with our private key for verification by other mail servers with our published public key in DNS TXT records.

I will *not* be going over verifying/filtering incoming mail. Only outgoing.

We'll be doing this on a Fedora Core 6 x86_64 machine.

Steps to great success...

• Install Postfix. You'll also need Perl, CPAN, OpenSSL and some luck.
  
  
• Upgrade CPAN - this is always good practice
  > cpan
  > install Bundle::CPAN
  
  
• Use CPAN and install the following modules
  > cpan
  # answer all the questions to set this up.
  > install Crypt::OpenSSL::RSA Mail::Address MIME::Base64 Net::DNS Net::Server Test::More
  
  You may have to force install Net::DNS. We had to.
  > force install Net::DNS
  
  
• Download dkfilter - alternatively follow the directions at dkfilter.
  > wget http://jason.long.name/dkfilter/dkfilter-0.11.tar.gz
  > tar xzf dkfilter-0.11.tar.gz
  > cd dkfilter-0.11
  > ./configure --prefix=/usr/local/dkfilter
  > make && make install
  > useradd dkfilter
  
  
• Generate your DomainKey
  > openssl genrsa -out private.key 1024
  #next line will generate the key published as a DNS TXT record.
  > openssl rsa -in private.key -pubout -out public.key
  
  
• You may have many domain keys. Each one is specified as a selector. For now we'll create a test selector. In your DNS records, add
  
  test._domainkey.yourdomain.com IN TXT "k=rsa; p=KYwekEKRO....OKzcWK; t=y"
  
  where p is the contents of public.key without the header/footer or newlines/spaces. Be careful not to cut off any characters on this step. The t refers to Test Mode. More information on DomainKey Implementation.
  
  Test your entry by using the 'dig' tool.
  
  > dig TXT test._domainkey.yourdomain.com
  
  You should see something along the lines of...
  
  ;; QUESTION SECTION:
  ;test._domainkey.yourdomain.com. IN TXT
  
  ;; ANSWER SECTION:
  test._domainkey.yourdomain.com. 7200 IN TXT "k=rsa\; p=MIGfMA0GC...9fOUedv02QIDAQAB\; t=y"
  
  If not... try again until you do.
  
  
• Startup the outgoing daemon.
  > /usr/local/dkfilter.out --keyfile=/your/private.key --selector=test --domain=yourdomain.com --method=nofws 127.0.0.1:10027 127.0.0.1:10028 &
  
  
• Configure postfix. In the master.cf file...
  
  under the smtp and/or submission line, add
  
  -o content_filter=dksign:[127.0.0.1]:10027
  
  and at the end of the file, declare the filter itself. I don't know what any of this crap means so a copy/paste might be prudent.
  
  dksign unix - - n - 10 smtp
  -o smtp_send_xforward_command=yes
  -o smtp_discard_ehlo_keywords=8bitmime
  
  127.0.0.1:10028 inet n - n - 10 smtpd
  -o content_filter=
  -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
  -o smtpd_helo_restrictions=
  -o smtpd_client_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o mynetworks=127.0.0.0/8
  -o smtpd_authorized_xforward_hosts=127.0.0.0/8
  
  restart postfix
  >service postfix restart
  
  
• Test it! There are autoresponders to test this stuff. An easy way is to just send email to Yahoo! Mail. When you get it, look at the Full Headers and you should see something like this...
  
  Authentication-Results: mta551.mail.mud.yahoo.com from=yourdomain.com; domainkeys=pass (ok)
  
  If it doesn't work, check your TXT record and make sure there aren't any line breaks or what not.
  
  
• When you're satified with the results, go ahead and turn test mode off by changing the TXT record from t=y to t=n. Add the dkfilter startup line to /etc/rc.local and you're good to go.
  
  
• DONE! Spam Email away!
  
  

References:
http://jason.long.name/dkfilter/

High Performance Reverse Proxy... Squid!

Little do you know but much of the internet world runs on a little known piece of software called Squid. While Apache, PHP, Ruby, Flash,.. etc may get all the glory, they would collapse on themselves without Squid.

Squid is a web caching server. In this article we will be using it as a reverse-proxy http accelerator for images. This can also apply to accelerating pretty much any static content from swfs, css files, file downloads, mp3s etc.. At HOTorNOT, we serve millions of static images via squid.

Choosing Hardware
We are choosing a very commodity "server" for this job.

• CPU - a well priced 3.4ghz Pentium D - squid tends to be light on cpu with most of the time spent in the interrupt loop. More processors doesn't tend to help any so a Xeon or Opteron would be overkill for this job.
  
• Memory - 4GB of 667mhz DDR2 - faster memory the better. the amount depends on how much stuff we're trying to accelerate. We want to serve most stuff from memory.
  
• Disk - a very commodity 7.2k rpm sata drive with NCQ (native command queuing). Seagate for the win here. Squid can be heavy on the disk in terms of concurrent random reads and it's best to choose something with SCSI-like command queuing to help prioritize based on where
  
• OS - Fedora 6 64bit. - A 32-bit OS is faster for this kind of job but we get limited by the amount of memory it can handle. For any machine with >= 4GB go for the 64bit.
• Squid - Squid 2.6
  

Compiling
Here's our compile time options and some explanation.

ulimit -n8192 #it's common for squid to eat all 1024 file handles so we'll increase it here.
./configure \
--prefix=/usr/local/squid \
--enable-async-io \ #fork reads into multiple threads. good for concurrency. only good for linux.
--enable-icmp \ #not really sure.
--enable-snmp \ #for SNMP monitoring.
--enable-htcp \ #for cache peering. only useful if you hav emore than 1 cache.
--enable-ssl \ #some images need to be SSL'ed for the user to have that nice lock icon in the browser.
--with-large-files \ #cuz that's the way she likes it.
--with-build-environment=POSIX_V6_LP64_OFF64 \ #need to set this or compile will die. you'd think it'd be automagic by now..
--with-maxfd=8192 #again, squid likes to eat file handles like a hungry hungry hippo hippo.

Configuring

We basically want this machine to sit in front of our webservers, cache, and serve images for them. So a few key points with the configuation.

#set port to be 80 for http, with the default domain pix.hotornot.com
http_port 80 accel defaultsite=pix.hotornot.com

#our parent "cache" is our webservers at pix.hotornot.com
cache_peer pix.hotornot.com parent 80 7 no-query originserver

#use up the memory.
cache_mem 3500 MB #use up that 4GB! almost...

#set the max size of an object (so you don't cache an abnormally large image)
maximum_object_size 500 KB
maximum_object_size_in_memory 500 KB

#aufs is for async-io, offload disk reads to a thread. big cache is always good.
cache_dir aufs /hot/tmp/squid_cache 50000 16 256

#don't cache 404's, always go to origin servers
negative_ttl 0 minutes


Optimizations

This is a pretty cool one. Whenever squid reads from a cached file, it has to update the access time on the file by writing to disk. This actually becomes quite slow with a busy image server so switch this off!

edit your /etc/fstab, find the mount entry where the squid cache is and next to 'defaults' add 'defaults,noatime'.

or do it on commandline!

> mount -o remount,noatime /your_partition


A cheap server like the one we have listed here can push > 30Mbit of static traffic no problem.

Database backup compression

Our first technical post is going to be about compressing text database backups. Insurance against the dreaded SQL injection attack or the power surge monster. If you'd rather skip all this incessant babble, scroll to the bottom for the conclusion.

We used to just use straight up gzip with no options. That seemed to work but what's really the best and how does that stack up with the latest/greatest stuff of the 21st century? (7z)

Well we decided it's about time to figure that part out. We tried this on a real database backup with real data. No better way is there?

How we got the backup in case you were wondering.

> mysqldump --opt --no-create-info --database ourdb

Now lets see how the compression stacks up. Remember that in a real company, we trade off space and computer power that it takes, we can't have the server doing dumps all night long.

Compression		Time	Size	Factor
none			3324MB
gzip		421s	993MB	29.9%
gzip -9		950s	974MB	29.3%
7z		7,876s	705MB	21.2%
7z -9		not	worth	testing
bzip2		1,417s	711MB	21.4%
bzip2 -9		1,531s	711MB	21.4%

Things to consider when looking at these numbers:

• Server will be offline for this period of time.
• This also affects slaving and the machine will have to "catch up" after.
• Does it have other duties?

So it looks like gzip is pretty good with just the defaults. A -9 flag seems to more than double the cpu usage for very little savings. 7z is just way out of the question with an insane amount of computing necessary. Bzip2 seems alright with default compression settings but is more than 3x the computing time...

Conclusion: Just use gzip, default options... the same crap we used before...

what a pointless experiment.

First post!

Hi from HOTorNOT operations. This blog is going to be used to publish technical findings about operations, scalability and general random operational voodoo. If you've ever wondered how a growing web company operates... read on!

Comments and criticisms are appreciated.