Fragmentation is a by-product of how the file system works. Ideally, all the data on a hard drive would be written sequentially, in a circular pattern on a platter on the drive, allowing for a direct, or bitwise, reading of the hard drive. When we delete a file, however, it may leave a gap in the smooth flow of data. Windows has no qualms about storing data in these gaps, even if it involves only putting part of a large file in the first gap it finds, another part in the next gap, etc. It’s sort of like trying to fit square pegs into a round hole.
When defragmentation reaches the point where dealing with it is causing too much movement of the read heads in the hard drive and really slowing down drive access, the disk needs to be defragged.
To be fair, Microsoft has gotten better and better at avoiding significant fragmentation, so the need to run defragmentation is much decreased in Vista and Windows 7.