For this purpose, I have been using a service called send to dropbox for a while until it started getting unreliable. Soon it didn’t upload any of the e-mails anymore without even supplying an error message and I ended up fiddling with my e-mail attachments by hand again. Not very funny.

So, I hacked together something that “works for me” (TM) and doesn’t rely on an external service. Well, in fact, I am still relying on Uberspace to host this, but (a) I could easily do this everywhere else and (b) it’s a huge improvement over send to dropbox which felt more like a blackbox.

This is how it’s done on Uberspace, but really the instructions are very much applicable to any Linux based setup.

Download and Setup Dropbox Uploader

Dropbox Uploader is an awesome little shell script written by Andrea Fabrizi which can be used to upload files to Dropbox from the command line. It is very handy since it has no dependencies except for curl. For my purpose, it needed a little tweaking since it would not accept files from standard input, so I forked it and am currently waiting if my pull request is being merged. I will update this article as soon as it does, since I have no interest in maintaining the fork. Andrea seems to be doing an awesome job with this.

So open a shell on your Uberspace and do the following:

mkdir -p ~/opt
cd ~/opt
git clone git://github.com/yeah/Dropbox-Uploader.git
mkdir ~/bin
ln -s ~/opt/Dropbox-Uploader/dropbox_uploader.sh ~/bin/

This installs the latest version of Dropbox Uploader in ~/opt/Dropbox-Uploader and creates a handy symlink to it in ~/bin which should be in your PATH already.

To verify if that last assumption is true (and to authenticate Dropbox Uploader), simply run

dropbox_uploader.sh

It should lead you through the process of creating your own Dropbox App and authenticating your copy of Dropbox Uploader against it. If asked for the Access Level I’d opt for App folder since this won’t give the app access to your entire Dropbox. Your credentials are stored in ~/.dropbox_uploader which is 600 by default, but if you’re a bit paranoid like me, App folder is still the safer bet.

If you’d like to test Dropbox Uploader, perform a simple upload like this:

echo "hello dropbox world?" | dropbox_uploader.sh upload - hello_world.txt

A file called hello_world.txt should magically appear in your Dropbox. Does it work? Yay! Then here comes the fun part…

Configuring qmail and reformime

Again, it should not be hard to adapt this to other MTAs, but since Uberspace uses qmail (or netqmail to be more specific) this tutorial is written for that.

At Uberspace, you get an unlimited number of e-mail addresses out of the box. Your primary is composed like this:

username@hostname.uberspace.de

Where username is your Uberspace username and hostname is the host your account is hosted on. What happens to e-mails coming this way is governed by a small file named ~/.qmail. In much the same way, you can use any e-mail address that follows this format:

username-foo@hostname.uberspace.de

Where foo can be anything you like. To specify what should happen with e-mails coming in via this address, you can create a file called ~/.qmail-foo.

So, for instance, if you want all e-mail PDF attachments sent to peter-dropbox@phoenix.uberspace.de to appear in your Dropbox, create a ~/.qmail-dropbox file with the following content:

| /usr/bin/reformime -X /bin/sh -c "if [ "\${FILENAME#*.}" == "pdf" ]; then ~/bin/dropbox_uploader.sh upload - \"\$FILENAME\";fi"

Yep. That’s one single line. It uses reformime to extract all file attachments and then uploads those that end in .pdf to your Dropbox using Dropbox Uploader.

Try it out. It should already work. There’s nothing else to do.

Doing a quick Google search yields results for the 1.2 version of pfSense which is outdated and does not use DynDNS hostnames for both ends, so I did a quick writeup of my own.

Prerequisites

First things first, create permanent hostnames for your pfSense and your FRITZ!Box. If your DSL provider has assigned permanent IP addresses, that’s fine. If they didn’t you’ll probably need something like DynDNS. Last time I checked, you could still get free accounts, otherwise it’s just a few bucks a year – probably a good investment. You’ll need to configure both the pfSense and the FRITZ!Box to update your DynDNS hosts whenever their IP address changes, but that’s pretty straight forward so I won’t cover it here. Fun fact: you can add CNAME records to your company domain pointing to your DynDNS host, so it looks even more professional. We use vpn.launchco.com for instance – how cool is that?

You’ll also need two different primary subnets for your networks, i.e. if your home network lives in 192.168.178.0/24, which is the standard network a FRITZ!Box uses, your work network has to use something else, like 192.168.1.0/24, which is by the way the standard that pfSense uses – so you should be safe if you’re like me a big fan of sticking with sensible vendor defaults.

Now, with the permanent hostnames and subnets in place, let’s get down to business.

Setting up pfSense

We’re using IPsec, so let’s head to VPN -> IPsec first and click the [+] icon on the bottom right to add a new phase 1 entry.

Fill the form in accordance to what you see on the following screenshot:

Obviously, replace your-fritz.dyndns.org with the permanent hostname assigned to your FRITZ!Box as well as your-pfsense.dyndns.org with the one on your pfSense box. The Pre-Shared Key should be a long random string. Don’t worry, you won’t have to remember it. You’ll just save that in the FRITZ!Box later and then you can forget about it.

Next up, we need a phase 2 entry. For that, click the [+] icon next to a label that says Show 0 Phase-2 entries and fill the form like below:

Here, you just need to make sure that you replace 192.168.178.0 with the actual subnet your FRITZ!Box uses. Again, if you’ve sticked with the default when setting up the box, this setting should be right for you.

That should be it for the pfSense. After saving it’ll probably ask you to apply or reload the configuration. This is safe to do now.

Setting up the FRITZ!Box

Now, let’s finish this by configuring a VPN entry in your FRITZ!Box. From my perspective, this part is much easier, because I’m just pasting code instead of fiddling with screenshots – yay!

Fire up your favorite text editor and paste the following code:

Make the necessary modifications according to the comments in the file. Then, open the FRITZ!Box configuration interface in your browser and head to Internet -> Freigaben -> VPN, use the browse button to select the file you just created and click on VPN-Einstellungen importieren.

That’s it – you’re done. In my first trials I had to go back to the pfSense interface and navigate to Status -> IPsec to click on a small [>] (“play”) button to get things rolling. Maybe you need this, maybe it just works without it.

Getting the connection up after a restart of either of the two routers sometimes fails which is most probably due to the fact that DynDNS updates have not yet propagated when the VPN tries to connect. In this case, just be patient; both boxes will keep retrying to open VPN connections and you can always stop/start on both ends yourself. Once a connection is made, the tunnels are usually stable and rock-solid. Enjoy!

However, this starts looking ugly rather quickly if you’re from Germany or France for instance, where use of umlauts and accents is very common. Something really nice like
J'ai montré les éléphants à ma sœur
becomes something really ugly like
j-ai-montr-les-l-phants-ma-s-ur.

So as Holger pointed out, I needed a diacritics table which I found here. After some modifications for the German language (e.g. ä -> ae, ß -> ss), I came up with this.

It’s still heavily based on what lehel built, so thank him, not me. I just wanted to put my improved version here, so I don’t forget it.

Update: I have created a Gist for this over at Github so we can continue to update it there…

Here’s how I built this for just under €500.00 using standard components and free open source software.

Selecting the hardware components

I found the HP ProLiant MicroServer (see Review and more Picures) to deliver great value for the price. At the time of writing, you can buy it for €209.90 if you’re in Germany like me.

The N36L (which I bought) comes with a single 250GB hard drive which obviously did not meet my “a lot of storage” requirement. So I bought 4 identical Seagate Barracuda Green 2000GB SATA drives which would add another €229.92 to the bill if you bought them today. I am not an expert in hard drives, but the Seagate Barracuda brand was familiar and “Green” sounds good as well.

If you don’t want your new server to host virtual machines at some point, you can probably get out your credit card and check out right now. If you’re like me though, you’d add another 2 bars of 4GB Kingston ValueRAM PC3-10667U CL9 (DDR3-1333) to your cart. The two of them together are just €44.24, so it’s no big deal anyways.

All components together will set you off €484.06. The rest is based on open source software (Debian mostly) which is free as in beer. More about that after the break.

I won’t explain how to actually assemble the components. Please read the manuals if you’ve never installed RAM or a hard drive. It’s pretty easy anyways. The HP even comes with screws and a sort of screw driver which can both be found in the inner side of the hard drive door.

Installing the base system

Now, I want my servers to run a proper operating system, so I went for Debian. You can most probably use Ubuntu or CentOS and get the same results. It’s just a matter of taste, I guess. To get things going, I downloaded a small image from debian.org, put it on a USB drive and booted up the HP.

Considerations

For the most part, the installation is pretty straight forward. You can set up everything from within the Debian installer. There are a couple of small gotchas to consider though:

• You’re going to need a very small (1MB should be largely enough) primary partition at the beginning of all disks. In order to handle the large 2TB disks, Debian is going to use GPT which in turn relies on that tiny partition.
• We’re going to encrypt the entire drive in order for your box to be really secure. As far as I know, Debian cannot boot from an encrypted partition, so you’ll need a small partition for the /boot filesystem. Personally, I’ve put that one on a RAID1 array, so I don’t need to worry about rebooting the machine in case of a disk failure.
• I remember having had a hard time booting from disk for the first time. For some reason, I had to set the proper boot device in the BIOS to make things work. Your mileage may vary.
• The HP has an on-board RAID controller. Usually, I am more comfortable with things I can fully control, so I went for software RAID which is built-in in Debian. Things might be a bit more performant using the hardware controller and you might still get the same level of robustness and security. I just didn’t use it, so keep that in mind when reading my instructions.

Step by Step

So, basically, here’s what you need to do to get your Debian up and running on your new box:

• On each drive, create the small (~ 1MB) GPT partition and flag it bios_grub.
• Again on each drive, create another slightly larger (~100MB) partition and flag it raid.
• Then, still on each drive, create a large (~2TB) partition taking up all the remaining space and flag it raid.
• Now, set up a RAID1 using the four smaller 100MB partitions, should be called /dev/md0.
• Then, set up a RAID5 using the four large 2TB partitions, should be called /dev/md1.
• Next, set up a crypted partition on top of the RAID5 (/dev/md1), should be called md1_crypt
• I recommend using LVM, so next, set up a physical volume on the crypted partition (md1_crypt) using up all its space.
• Now, create a number of logical volumes. I’d recommend to set up one each for the usual suspects /, /home, /var, and /opt at least, but again, your mileage may vary. The nice thing about LVM is that you can add and change logical volumes at any time.
• Finally set the moint point of /boot to your RAID1 (/dev/md0) and continue the installation. I’ll leave the package selection entirely up to you. Usually, I like to install a bare minimum during setup and get more stuff as I need it later on.
• If everything went well, your new HP should boot up a couple of minutes later, asking for the password for your crypted disk and present you with the initial login prompt.
• I usually start by installing ssh and sudo which enables me to do everything else from my own workstation. But you can continue the setup in your closet or wherever your HP sits, no problem.

AFP and Apple Time Machine

Disclaimer: The following is based on a great article by Chris Boot. I read it there first.

Enabling Time Machine support is actually much easier than I first thought and it doesn’t even require serious hacking, compiling or even using non-standard repositories if you’re on Debian. Here’s what you have to do:

• If you plan on using Time Machine with Lion, you’ll need a newer version of netatalk than is currently available in Debian Squeeze, so you’ll have to add Wheezy to your /etc/apt/sources.list. Edit the file and add the following lines at the bottom:

  # testing
  deb http://ftp2.de.debian.org/debian/ wheezy main
  deb-src http://ftp2.de.debian.org/debian/ wheezy main
  deb http://security.debian.org/ wheezy/updates main
  deb-src http://security.debian.org/ wheezy/updates main

• Also create a file called /etc/apt/preferences.d/00pinning and add the following lines:

  Package: *
  Pin: release a=stable
  Pin-Priority: 700
  
  Package: *
  Pin: release a=testing
  Pin-Priority: 650

• Get Netatalk and Avahi using apt: apt-get update ; apt-get install -t wheezy netatalk avahi-daemon
• In order to export a share using Netatalk, edit /etc/netatalk/AppleVolumes.default and add a line that reads /srv/timemachine "Time Machine" options:tm and be sure to create /srv/timemachine or change it to something you want to use as a location for your shared folder.
• Next, edit the file /etc/netatalk/afpd.conf and add - -tcp -noddp -uamlist uams_dhx.so,uams_dhx2.so -savepassword -nodebug -icon at the bottom. It will configure netatalk to use the authentication methods required by Lion and some other stuff I found useful.
• Now, doing a /etc/init.d/netatalk restart is actually already enough to export the folder using AFP ready for Time Machine goodness.
• However, if you’re an Apple user you probably want to see your new server in the finder instead of just accessing it via its IP address. Enter Avahi. Create the file /etc/avahi/services/afpd.service and paste the following code:

  <?xml version="1.0" standalone='no'?><!--*-nxml-*-->
  <!DOCTYPE service-group SYSTEM "avahi-service.dtd">
  <service-group>
  <name replace-wildcards="yes">%h</name>
  
  <service>
   <type>_afpovertcp._tcp</type>
   <port>548</port>
  </service>
  
  <service>
   <type>_device-info._tcp</type>
   <port>0</port>
   <txt-record>model=Xserve</txt-record>
  </service>
  
  <service>
   <type>_adisk._tcp</type>
   <port>9</port>
   <txt-record>sys=waMA=YOUR-MAC-ADDRESS,adVF=0x100</txt-record>
   <txt-record>dk0=adVF=0x83,adVN=Time Machine</txt-record>
  </service>
  
  </service-group>

  Be sure to replace YOUR-MAC-ADDRESS with the actual MAC address of your HP box and to use the same string for Time Machine as you used in AppleVolumes.default.

And that’s it. Your Mac computers should start seeing your HP box and you should be able to use it as a time capsule for your wireless backups.

Please note: I wrote this guide a couple of weeks after I set this up at the LAUNCH/CO office. So most stuff is from memory. If you find any mistakes or happen to get stuck somewhere in the process, please leave a comment and I’ll try to help.

Update: You may have noticed that your shiny new DIY time capsule has stopped working since you upgraded your Macs to Lion. This blog post looks promising – I will check it out and update this post as soon as I find the time.

Update 2: If you have set up your N36L using these instructions before August 13th, 2011, your box won’t work with OS X Lion. I have just upgraded the instructions to use the latest version of netatalk. You should be able to make Time Machine support work again by following the new AFP and Apple Time Machine instructions adapting your existing config.

[mysqld]
default-character-set=utf8
default-collation=utf8_unicode_ci

However, if for some reason you didn’t do that and have used software which hasn’t been consistently explicit about character sets and collations, you end up with a nice mess of character sets and collations.

There is a great post on serverfault which helps you out. It comes down to one command which will take some time based on the size of your database:

mysql -B -N --user=user --password=secret -e "SELECT DISTINCT \
CONCAT( 'ALTER TABLE \`', TABLE_SCHEMA, '\`.\`', TABLE_NAME, '\` CONVERT \
TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;' ) FROM \
information_schema.COLUMNS WHERE TABLE_SCHEMA != 'information_schema';" \
| mysql --user=user --password=secret

Update:

And of course you need to alter the defaults for existing databases as well:

mysql -B -N --user=user --password=secret -e "SELECT DISTINCT \
CONCAT( 'ALTER SCHEMA \`', SCHEMA_NAME, '\` CHARACTER SET utf8 COLLATE \
utf8_unicode_ci;' ) FROM information_schema.SCHEMATA where SCHEMA_NAME \
!= 'information_schema';" | mysql --user=user --password=secret

If you’re using Debian, the APT sources will save you some headaches. Just add

deb http://apt.opscode.com/ squeeze main

to your /etc/apt/sources.list and do something like

wget -qO - http://apt.opscode.com/packages@opscode.com.gpg.key | sudo apt-key add -
sudo apt-get update

to be sure to be getting what you asked for. Then, a little sudo apt-get install chef will do the trick to set up a client and sudo apt-get install chef-server will supercharge your node with a fully blown chef server, including but not limited to CouchDB, Solr, RabbitMQ and other fancy stuff. (You’ll want to do this on two different nodes, so use Virtual Box or something.)

After you’ve set up two nodes like that, try following the rest of the instructions in this tutorial and do the first cookbook example, then you’ll have come as far as we have today.

I will update this post as we dig deeper – hopefully later this week.

From a technical point of view, an affiliate program is nothing fancy at first glance:

1. give your customer a link with a unique token
2. once a visitor signs up, check if a token is present and look up the respective customer
3. reward them!

However! There’s some technical pitfalls.

Keep track of the token

This is a rather easy one: of course, you have to remember the token throughout entire visits. You can’t expect a visitor to turn into a paying customer right on the first page. They will check out your site, visit a couple of pages, and maybe even come back another day to buy your product. You still want to reward your affiliate, so cookies will be your single option.

Don’t mess with Google

We’ve learned this the hard way with Magpie and it took us quite some time to recover our page rank, so be sure to read this! Google does not like duplicate content. If you’re copying what others write on the Web or if you have a lot of pages with similar or even identical content, Google’s algorithms will classify your site as spam. What does this have to do with your affiliate program? Well, all those referral links are different because of the token, yet they will most certainly render the same content.

So what can you do? Redirect. Don’t let your app render a page if the request URI contains an affiliate token. Redirect to the actual page using status code 301 (moved permanently). This way, Google will know that the link is still valid (and thus you will get most of the link juice from referring sites), but that its location has changed.

How to implement?

For a long time, we did this within in our application. Rails makes it really easy using before_filter, so it’s no big deal. However, your setup may be more complex. Maybe you have multiple apps on subdomains or sub-URIs and maybe they run on different frameworks. Just think of your corporate blog, most of the time it’s a WordPress. But you’d still want to reward your affiliates if they send you traffic via a link to a great blog post you’ve written, right?

For Planio, we moved the redirection and cookie part to the Web server. Below is a short and sweet Apache config snippet which works really well for us:

# affiliate cookie
RewriteCond %{QUERY_STRING} (.*)ref=([a-zA-Z0-9]{6})(&(.*))?
RewriteRule ^(.*)$ $1?%1%4 [CO=affiliate:%2:.plan.io:43200,R=301,L]

It does everything for us, so our apps don’t have to worry:

• detect a token in a request URI (we use a ref= query param with a 6 character token)
• set a cookie named affiliate using the token value which is valid for all our subdomains and for 30 days
• redirect to the same page using 301, removing the ref parameter and keeping all other query parameters intact (this is great for other tracking stuff, like the params you can generate for Google Analytics)

In the end, we just need a one-liner in our signup code that reads the cookie, finds the affiliate and associates the affiliate with the newly created account.

Update: Thomas points out that you could tell Google to ignore certain query params and avoid 301 redirects using canonicals. He also claims that Google would be my friend. Not so sure about the last one, though

I hope this was useful to you. Do you run affiliate programs for your products? What are your experiences? How did you implement them? I look forward to your thoughts in the comments!