Microsoft Live Search In-Depth Review – Part 2

In Part 1, we looked at how Live Search performed with a two word query. Such wide open searches really don’t tell you too much about the quality of the search engine since the query didn’t really tell the search engine anything about what you really wanted. Let’s look at how Live Search performs with some more intelligent searches.

Next, we searched for comodo antivirus reviews without quotes. We’re looking for Live Search’s ability to bring us back results about what people think about Comodo Antivirus. We’ll be looking for the same ability to handle the query as in Part 1, but we’ll also be looking at how it handles reviews. Is it smart enough to give us a page with the singular version review? Is it easily tricked by pages that have the word review, but are obviously not reviews? Let’s find out.

Live Search falters a bit here. Virtually the whole first page of results are from download sites where you can get the product, but for most of these sites, the "reviews" are the two and three sentence blurbs left by users. We can’t really call that a FAIL, but it isn’t great either.

Result #5 is the highest ranked result that is a page that is actually a full review and not user comments. These results might suggest that Live Search is overweighting key words instead of actual content.

Did you notice result #7? That is a nice full review here on Bright Hub, so kudos to Live Search there. (Just kidding.)

What is particularly disappointing is the significant difference in the quality of results we get by searching for comodo antivirus review instead of comodo antivirus reviews. In the graphic at the right, the starred results are the ones that contain a full review. We don't make any judgement on the quality of those sites or their reviews, just that they have a complete review, and not just user comments that are labeled as reviews.

Keep in mind that the sites with all of the user reviews are to have the text, “user reviews” whereas the sites that have a single site-authored review that they personally stand behind are likely to say something more like, “Comodo Antivirus Review”. Semantics? Maybe, but users looking for reviews of a product are more likely to type reviews than review. Which means that users are less likely, in this case, to get what they are looking for by doing what comes naturally.

Specific Live Search Results Control Experiment

Comodo Antivirus hasn’t been out of beta very long and isn’t very well known, so there are not a lot of review pages out there. That is exactly why we used it as our test term. But, how does Live Search perform on something a little more widespread?

We searched on best antivirus software to find out. The number of results jumped to over 90 million. No excuse to not have a nice solid list of useful search results here. So, how did Live Search do?

On the first page of results, we see just one entry that would be considered not a reasonable result from our query. That site has the words on it, but most definitely is not about what we are looking for with that query. All of the others match the query just fine. However, one glitch does raise its head. Two of the results are quite old. In fact, one is from a website which proudly states its last update time…it's from 2004!

Which brings us to Microsoft Live Search In Depth Review Part 3 – Options and Features