Your options aren't just limited to external spinning disk drives, and these other varieties have their own advantages as well.
SSDs or solid state drives are a popular choice for portable storage, for instance. Because there are no moving parts on them, they require no external power, no fan to keep them from overheating, are more rugged and are thus less likely to break... The list goes on. The only real disadvantage to SSDs is the write problem, where they will slow down and become more corrupt over time. Flash drives, one of the smallest, most convenient portable storage devices out there, are a type of SSD.
Of course, you could just upgrade your internal hard drive for something bigger, if more space is all you need. Internal hard drives tend to be cheaper and more stable for the same amount of space, though this is good neither for back up nor for portable storage needs.
CDs and DVDs are another, often overlooked piece of technology for memory storage. Incredibly cheap and fairly small, many people still opt for these rewritable disks. While many people find it a pain to insert, read, write and eject every time they want to check out another few megabytes of data from their archives, it is a sound form of back up, and one that can last a long time if properly protected.