The first thing you should notice is the anonymity. Even if you hover over the shortened URL, you don't know where you're really going, which makes you an easy target for spammers and malicious sites. More than once, I've clicked on tweets that had an interesting tag and a shortened URL that led to an affiliate marketing site that had nothing whatsoever to do with the content of the tweet. From a user standpoint, this represents unknown dangers, which can be significant for the uneducated, and unprotected, casual web user. In addition, there's always the possibility of hijacking if the url shortener service decides to try and make money from your clicks by adding an advertisement between your URL and the destination. So far, that hasn't happened, but it could. They can also be used to track user movements across the web. Maybe it sounds paranoid, but ad networks can already do it, so why not?
Another consideration is the value of linkbacks. Linkbacks are valuable SEO currency that help bring a website to the top of the search page. Url shorteners turn golden linkbacks into mush. No matter how many people click, the originating website doesn't get the credit. It also makes tracking down original sources more difficult, especially if the link becomes corrupted or broken.
The last thing to consider is the added layer of transit in the chain of communication. It makes the underpinnings of the link structure that much more complicated with an additional lookup operation.
