So, these are pretty obviously powerful pieces of technology, and one that is evolving quickly. The single best thing that you can do to address privacy concerns on smart cards is simply to know what's going on with regards to smart card policy, technology and applications. Awareness is powerful!
There are three main applications of smart cards: financial services, ID services, and public transportation, each with unique security issues associated with them.
Financially oriented smart cards have been rolling gradually into the system for several years now. They are largely held to be more secure than non-smart cards—think credit cards with a lot more anti-forgery support built in.
Government ID services have been looking at smart cards for some time now as a way to keep better track of citizens to fight everything from terrorism to illegal immigration. With all the data placed together, as opposed to sprawled out amongst agencies at the local, state and federal level, it could be much more difficult to hide in the shadows. This has a very deep potential for abuse, though, particularly if the information got into the hands of a malicious third party.
Public transportation is not the most conspicuous use for smart cards, but it's a growing market. The most notable use is the Oyster Cards used in London public transportation system. These cards could theoretically be used to track an individual's movement throughout the city by either the operator of the public transit or the government, which presents a privacy concern. For example, this has already been an issue in the UK, where M15 wanted to use this information to track terrorists, even the the card has already been cracked.
Health information is another concern. While smart cards with health information is not all that common yet, it's another use that's growing, and that's certainly sensitive information that could potentially be abused by third parties.
Now, mostly there are different smart cards for different applications. However, many people are frightened at the prospect of having a single card for all of them—a combined national ID card, credit card, biometric, driver's license, anything you can think of, in a single square of plastic and circuitry. The wonderful usefulness of such a universal card is only matched by the frightening potential for abuse. Instead of having to individually track down each item to take advantage of someone, you instead of a single piece of technology that contains someone's entire life.
Right now, smart cards have no real standards, varying from company to company, and government policy progress on unifying forms of identification has been markedly slow worldwide. The only real success story has been Malaysia's MyKad program launched in 2001, and even that has been greatly underused.
So, there's no immediate threat, if you find this concept to be uncomfortable, but the possibility is there.
This is a constantly evolving field of both policy and technology, so keeping an eye on the news for changes in trends is, again, the best thing you can do to address any privacy concerns you might have.