Find out when your server's peak hours are, and check your server's time zone against that of the servers in your battle group. You're much more likely to encounter casuals who are killing time with a few matches if you're playing late on a Saturday afternoon or on a holiday weekend.
If you're queueing at 3:30 a.m. on a Wednesday night, you're asking to get stomped. The only people playing at that hour are insomniacs, day-shift workers, and obsessive, ultra-serious PvP hardcores who can't wait an extra day to inflate their rating for that new mace or belt.
Back in the day, you could get a sense for who was queueing -- at least on your own server -- by standing in front of the battlemasters in Shattrath, Orgrimmar or Ironforge. The advent of queue-from-anywhere PvP has dispersed those crowds, but you can still get a feel for how populated the queues are by testing the waters.
If you're waiting five minutes between matches and running into the same tough combos after a few matches, it's time to give it a rest and come back when the queue is more active.