Rights in the US
The US provides photographers with a healthy dose of freedom, as guaranteed by the 1st Amendment Right of free speech. However, there are a few twists here and there. The following is assuming that you do not have explicit permission to take a picture of the subject:
You may take pictures while on public property. You may take pictures from public property into private property. Not even just that: even places that are open to the public, even if they are on private property, are OK. Malls, office buildings, all fair game. Cool, huh? The issue in the US comes down primarily to the subject's expectation of privacy.
One caveat here: military institutions. These generally may not be photographed without some pretty high-level permission, and it's not a good idea to test this. This is a direct security concern. That being said, any other security concern expressed by other individuals provides no ground whatsoever for cessation of photography. Vague 9/11 security concerns are not justification for ceasing photography. This includes most subways and railways, though again, some state restrictions may apply.
Now, publishing a photograph is whole other can of worms. As a photographer, you cannot publish photographers that reveal private facts about any individual or otherwise violate their intellectual property, even if it was taken given the previous criteria. This is especially true if it would offend the subject, or is not newsworthy or of public concern. How you frame the photo in its publication is also an issue, as if it puts the subject in an offensively false light, you can be sued for libel. Finally, using such a subject for purely commercial purposes, such as for an ad in a newspaper, as opposed to journalistic or artistic purposes, is not technically acceptable, especially where trademarked or copyrighted images come into play. These are all civil concerns, and not criminal.
That being said, if someone demands confiscation of your camera, you are under no obligation to give it to them. Period. The only caveat is when forced to do so by a court order, but that obviously can't be done by a police officer on the scene. If they try to threaten you? Well, bluntly, you're the one who can sue them for theft or coercion.
For an excellent laymen's overview of US photographer's rights, check out this article from USA Today.
For an article going through the legal basis for these rights, check out this free online book.
For a state-by-state overview of specific photography laws pertaining to them, check out this index.
For a print-out overview of your rights as a photographer, check out this site.
Still want more information? Check out this ever-growing list of resources with regards to photographer's rights.