Setting Up A VPN Server On OSX 10.6

Setting Up A VPN Server On OSX 10.6

MAY 31, 2010 BY ANDREW MILNE

I’ve recently setup a VPN server on my OSX box so that I can connect my iPhone and iPad to my home network securely.  This lets me use tools like Air Video (to stream my video collection to my iPhone) without having to make them available to the internet as a whole.  It may also let me view sites like BBC iPlayer when on holiday abroad (though that is still to be tested).

OSX ships with a VPN server but the configuration GUI is only present in the server edition of the operating system.  Some digging around shows that everything you need is already installed and waiting to be configured.  There is a shareware tool called iVPN that offers to do the configuration job for you at the low low cost of $15.  There is also an older version of iVPN that’s open source. Something sits wrong with me about these tools that I can’t quite place so I decided to do it from first principles. Most of this information comes from a Mac OSX Hints page (big thanks) and from searching the web.

 

Steps:

  1. An Introduction
  2. Store a secret key in the OSX Key Chain
  3. Configure the VPND service
  4. Set up launchd to start the vpnd service at startup
  5. Configure the Firewall
  6. Configure the iPhone

 

Step 1: An Introduction

There are lots and lots of different types of VPN setup and I honestly don’t understand how most of them work.  I do know that we will be using the L2TP protocol.

The phone will need 3 things to connect to the VPN server on the mac; a user name, password and a shared secret.  The user name and password correspond to an account on the local computer.  The shared secret is a code known only to the server and client and is used to secure the connection.

Were going to do a lot of tasks on the command line as the root user so start up the OSX terminal and enter the command:

and give it your password when it asks.

 

Step 2: Store a secret key in the OSX Key Chain

The shared key will be stored in the OSX Key Chain, this puts it some place secure rather than storing it in plain text where it can be seen by anyone with access to the box.

Ideally the shared key should be complex and hard to guess.  Personally I use a 64 character random hexadecimal key from https://www.grc.com/passwords.htm but you may want to use something a little less awkward to type in.

 

Added note by Chris: another password generator tool available in here – https://www.comparitech.com/privacy-security-tools/password-generator/

 

To store this run the command:

Replace “shared key” with whatever shared key you picked above.

 

The VPN server is two part.  The actual server is called vpnd but there is a second task called racoon.  Racoon is, I believe, responsible for setting up  the initial connection and handling the security.  The “-T” option in the above command gives racoon permission to access the keychain and read the value

 

Step 3: Configure the VPND service

VPND takes it configuration from a standard plist configuration file.  Start up vi (or the editor of your choice) and edit the file:

The file content should be:

There are three values above that you need to set for your own network:

  • Set the value marked XXX.XXX.XXX.XXX to the IP address of the server.  If you have more than 1 network interface set it to the one you want the server to listen on (e.g 192.168.2.10).
  • The values YYY.YYY.YYY.YYY and ZZZ.ZZZ.ZZZ.ZZZ indicate the range of IP addresses the VPN server should assign to clients when they connect.  Make sure this range isn’t in use by any other computers or DHCP servers and its big enough for the number of clients you want to connect. (e.g 192.168.2.100 and 192.168.2.120).
  • It’s important the file has the correct permissions:

 

 

Step 4: Set up launchd to start the vpnd service at startup

We need to make sure the vpnd server starts up each time we restart the computer, doing it manually would get boring quickly.

Starting boot tasks is handled on OSX by the launchd service.  Create a new plist file using vi (or your editor of choice) at:

Put in the following content:

It’s important the file has the correct permissions:

There is two ways to get this file to read in and the server to start.  You can reboot your computer or you can issue the following command:

You should now have a running vpnd fully configured and ready to connect to.  We can check this by examining the log files:

This file should contain lines of the form:

If it doesn’t your going to need to start doing some debugging.  Check the contents of the /var/log/ppp/vpnd.log or /var/log/system.log for useful messages.  The comments on the Mac OSX Hints page have a lot of useful information on things that could go wrong.

 

Step 5: Configure the Firewall

Make sure that your firewall / router is configured to forward UDP on ports 500, 1701 and 4500 to the server box.

There are so many different routers out there that you’ll need to go read the manual or search online for how to setup your specific brand.

 

Step 6: Configure the iPhone

If everything above went well you should now have a fully running and secured VPN server that can be accessed from any place on the internet.

To set your iPhone up to use the server go through the following steps:

  1. Open the settings app
  2. Select “General” > “Network” > “VPN”
  3. Add a new VPN configuration
  4. Set the VPN type to L2TP
  5. Configure the following settings:
    • Description: Anything you want
    • Server: The IP Address of your server (This is the public address given to you by your internet provider.  Depending on your provider this address may change frequently.  I recommend setting up a dns alias account with http://www.dyndns.com/ to make this step easier and more robust)
    • Account: The user name of an account on the server (this can be the one you normally log in as)
    • RSA SecureID: Off
    • Password: The password for the account you set above
    • Secret: The shared secret you picked above (enjoy typing in the 64 character hex key if you used it. It’s worth it!)
    • Send All Traffic: Yes
  6. Turn the VPN connection on via the switch at the top of the “General” > “Network” > “VPN” page.  A switch also appears near the top of the launch screen of the settings application
  7. Once your connected you should see a blue “VPN” icon in the bar at the top of the iPhone screen

Some of these settings could use going over in more detail.  The VPN connection uses two levels of protection.  The first is a user name and password that can be used to log on to the server machine, you can use your normal user account or create a new one with less permissions.  The second is the shared key, which wraps up the entire communication.  The longer and more complex your shared key is the harder it will be to break.

 

The “Send All Traffic” option tells the iPhone to send all traffic over the VPN connection, not just traffic directed at the VPN server.  You want this on as it protects all of your traffic to any site by encrypting it and sending it to your VPN server before it then makes it out on to the internet.  This makes it almost impossible for someone to monitor what your doing when your on a public WIFI or using 3G.  It also has the effect of making your public IP address appear to be that of your home internet connection, in theory this lets you use UK restricted web sites when you’re out of the county (iPlayer etc.), but it may not work if the site uses more than just IP to determine where you are.

 

If you have any problems check the /var/log/ppp/vpnd.log or /var/log/system.log files for useful messages. The comments on the Mac OSX Hints page have a lot of useful information on things that could go wrong (keep an eye out for the dreaded MD5CHAP error that seemed to plague people on older versions of OSX, though I didn’t see it on 10.6).

 

Hopefully that’s you now up and running.