How to catch Linux system intruders

There's no doubt that Linux is a secure operating system. However, nothing is perfect. Millions of lines of code are churned through the kernel every second and it only takes a single programming mistake to open a door into the operating system. If that line of code happens to face the Internet, that's a backdoor to your server.

The eyes have it

Patches are often released quicker and more often for Linux systems than they are for proprietary products. This is because anyone can take a look at the code – and thousands of developers regularly do. This 'many eyes' approach, which was advanced by Linus Torvalds, means that problems like back doors are usually found quickly. It also means that fixes are released just as quickly.

There's been a very vocal debate on the Linux kernel mailing list recently, with developers arguing about whether security bugs that are fixed should be formally announced and documented. Torvalds believes that making a big song and dance about security patches attracts the attention of miscreants like bees to honey. Others believe that anything but complete openness goes against the philosophy of Free Software.

The debate is still burning, but while the overlords of the Linux kernel are deciding which way to swing, there's a great deal you can do to protect your system, even if an Internet facing service is compromised in some way.

Watching for traps with Tripwire

Tripwire generates checksums from all the essential binaries that are running on your system. Every file has a completely unique checksum. If a single bit of data changes in the file, the checksum it generates will be completely different.

These checksums can't be cracked or duplicated because they rely on a tried and tested method of encryption. Using this database of checksums, Tripwire will periodically recheck the value of each binary file in its database. If any file has changed, the wire is tripped and an alarm is signalled.

This system is incredibly effective, because the first thing a hacker usually does when they gain access to your system is replace important system files with their own version. This is known as a 'rootkit', and it means that the hacker can always access your system – even after you locate and update the original 'back door' security problem.

There are various tools that look for the tell-tale signature of a rootkit installation, but Tripwire pre-empts these tools by catching changes to the file system – hopefully before the rootkit can even be used. For this reason, you can find Tripwire in the package repository of nearly every Linux distribution we can think of. Just search for it and install.

Tripwire defence has been so successful that a proprietary company has been launched on the back of the original Tripwire product; both are confusingly called Tripwire. Thankfully, however, the opensource version of the software is still popular, despite the fact that, like all Linux security tools, it takes a considerable effort to get it working.

Most of that effort is to do with editing arcane configuration files. Choosing Ubuntu or Debian can help with this, as both of these distributions include a quick setup wizard when you install the packages. We've used Ubuntu Hardy Heron in the example configuration below. We've also provided the manual commands for configuration if you're using a different distribution.

Make sure everything that needs to be installed is installed. If you add more packages after the Tripwire installation, you'll have to go through a rather convoluted reconfiguration routine. For this reason, it's more straightforward to install Tripwire after you've configured and set up the server exactly as you need it.

Configuration

After installing the Tripwire packages, you'll be presented with the Tripwire configuration wizard. The first question asks whether you want to create or use your site passkey. This is the unique encryption key that's used to generate the checksum information for your files, and Tripwire uses two – one for files that are only going to be used on the installation machine (such as the Tripwire configuration files themselves), and another for files that may be accessed and used on other machines across a network. The next page of the wizard will ask you for these.

However, there is a caveat. If someone else has a copy of the same keys, they can potentially create files that appear authentic. There's a slight security risk with keys generated by the Debian installer, so if you're in any doubt about the security of your system, you should generate the keys manually (see the next paragraph for how to do this). Otherwise, continue with the next page of the wizard, enter long phrases that you can easily remember for both configuration keys and let the set-up wizard generate the configuration file.