Of course, some older computers can't boot from USB, which makes booting Ubuntu from your external hard drive a bit of a problem. This is an issue with the BIOS being out of date which, while not easily updateable, can be done after a lengthy process.
Not interested in updating the BIOS? Get a boot medium. Basically, a boot medium is an alternate boot loader on a CD, floppy, or some other sort of medium that that that version of BIOS supports. You can place GRUB or some other bootloader than does support USB on it. So, essentially, the boot medium is booted by the computer so that you can boot the USB device. This article gives you the details.
Keep in mind that older computers may have trouble running the latest, more CPU-intensive versions of Ubuntu, though proprietary operating systems like Apple or Microsoft of course present even more difficulties. Also, finding the correct drivers for older computers may present a bit of a barrier, though not an impossible one.