When scientists go on the hunt for cancer genes in dogs, they are looking for specific genetic markers. These are small DNA sequences that differ between dogs, and their chromosomal locations are known about. If these markers are consistently seen in dogs with cancer and not in healthy dogs, it gives a strong hint that there is a cancer gene nearby.
This sort of genetics research is also carried out in humans, but what makes the dog so valuable is that to carry out effective and meaningful research you need to have generations of afflicted families and a wide DNA sample set. Dogs have shorter generations and many more offspring than humans and so are ideal subjects for this kind of research.