The biggest thing you can do to take your photos inconspicuously? Take your pictures as quickly as you can. Before you whip out your camera, decide how you want to compose the image and get in position. This will save a lot of time.
Another thing you can do is to have your camera settings set ahead of time. If you can save modes on your camera, perhaps one for shots in the sunlight, another for shadows, another for mixed lighting, or whatever mix of settings you'll be encountering, then that will save fiddling time. You'll have to tweak a few things once the camera is out, of course, but knowing exactly what you need to do will shave time off of that as well. Think before you shoot!
Many cameras have rotatable LCD screens, meaning that you can shoot from a variety of positions without having it up near your face, which is a very conspicuous place to shoot from. Shooting from your waist may take a little practice, but take a good shot as you walk by without pause is a good skill to have under your belt, especially for street photographers.