External hard drives that connect to your PC via a USB port are a good choice for backup, so long as you accept that they'll be slower than an internal drive.
However, if you happen to have any old internal IDE or SATA drives hanging around there's a much cheaper solution.
This device converts your old disk into an external USB drive. Just configure your disk as a master, connect up the power and data cables and screw the disk into the case to create an external hard disk.
This is a good solution for laptops, but for a desktop PC an alternative is the 'mobile rack', available from the same supplier for £7.75. This turns the disk into a removable internal drive that operates at its full native speed and with no limit on the disk capacity.
In reality, old disks will often be a bit smaller than the disk in your PC, but that probably won't matter unless you want to make disk images.