Professional photographers regularly face angry people who argue against their right to take a photo. It may surprise many with privacy concerns, but they probably can't do anything about it. Photographers' rights are fairly broad.
Photographers' rights vary all over the world. This guide focuses on the United States, but laws are largely the same in the United Kingdom, Canada and Australia. Of course, dictatorships all over the world impose ever-changing, draconian and subjective laws.
Much of the U.S. information is adapted from a handy guide provided by Oregon attorney Bert P. Krages, and applies to video and still photography rights. It should not be considered professional, legal advice. The umbrella rule for U.S. photographers' rights is that anyone can take pictures of anything in any public place, including sidewalks, streets and public parks. That includes taking pictures into a private place from a public place. Photographers can take pictures inside a house from the sidewalk. For the most part, these photos can also legally be published and sold.
This is a broad rule that covers photographers' rights to capture images of children, infrastructure, airports (from the outside), criminal activities and celebrities. Shooting and publishing photos of children in public is legal, though sexually explicit photos may run afoul of the law. Also, even heightened national security concerns do not allow a law enforcement agent to infringe on photographers rights around bridges, water towers or other alleged terrorist targets.
Photographers' rights include images of law enforcement officers and crime scenes, although police officers and others may legally ask photographers to move out of harm's way or into an area that does not directly interfere with their work. Businesses have rights to protect their trade secrets by blocking photographers, but only inside the establishment. Photographers' rights extend to taking pictures of a display window or other objects visible from the street or sidewalk.
As noted above, photographers' rights largely do not extend to private property. Generally, a photographer needs permission from a property owner. Of course, photographers can always try to follow the adage of "It's easier to beg for forgiveness that to ask for permission." People can reasonably expect privacy in some locations, such as a restroom or hospital room, even if they are technically public.
There are also legal, governmental restrictions on photographers' rights at military installations, schools, prisons, sensitive research facilities and other places that are publicly owned but not universally accessible. Of course, you may see the inside of one of those prisons if you try to take photos at NORAD or Area 51.
That said, threats and harassment by a photographer's subject may constitute a crime. Without a court order, nobody may confiscate a photographer's film, except a police officer in the course of an arrest.
Certainly, photographers' rights should be tempered by common sense. For example, a woman might ask you not to use a photo of her child for fear of her abusive ex-husband seeing the photo and tracking her down. Legally, you have no obligation to honor the request, but it's a reasonable accommodation for her safety.
Finally, remember that you're already on camera. Corporate and government video cameras are everywhere. The New York Times reported in 2007 that an estimated 3,000 video surveillance cameras are running in lower Manhattan. Smile, you're on everyone's camera.