SERFs Up!

By Aaron Goldman, VP of Marketing & Strategic Partnerships

Starting today, Resolution Media will be rolling out SERF -- Search Engine Rotation Fridays. And we invite the entire search marketing community to join us.

Some of you may already be familiar with this concept. Until now, it’s been known as Google-Free Fridays. Here at RM, though, we want to be PC to all our search engine partners so we’re going with SERF.

Here’s how it works. Each Friday, we’ll pick a different search engine to use as our default for the day. Then we’ll report back the following week on anything interesting we learned and how it might be applied to search marketing programs.

Why are we reviving this tradition? Well, it started with a lunch we had a few weeks ago with some folks from Microsoft. It was right around the time that Cuil launched and we were debating the merits of launching a new engine with a huge PR push before working out all the algorithm’s kinks.

The talk quickly turned to what it would take for any new (or current) engine to match Google in terms of relevancy and scale. Not surprisingly, the Microsoft folks insisted that Live Search was already there and, if more people tried it, they’d realize it and kick their Google habit.

I piped in saying that the times I’ve used Live Search I didn’t find the results as relevant and that’s why Google is my default search engine. Patrick Harris, National Sales Manager, Search Agency Sales, for Microsoft Advertising, took exception to my comment. He asked me if I’d ever used Live Search and Live Search only for an entire day. I told him I had not but I frequently used it to see alternative results when I can’t find what I’m looking for on Google (not to mention check out new features that could be leveraged for our clients.)

Patrick responded that spot-searches are not the best way to gauge the efficacy of a search engine. After all, if Google didn’t return what I was looking for, did I really expect Live Search to? Or was I just performing a difficult query?

This point really rang true with me. I recalled Danny Sullivan’s review of Cuil in which he submitted 9 queries in an effort to evaluate its relevancy. Danny (and Patrick) emphasized that 9 queries does not a review make.

The truth is, it’s all those little queries that pop up throughout the day that are the true test of a search engine -- the quick dictionary definitions, the map directions, the navigational queries, etc. Those are the much more popular queries that the engines all have different approaches to and thorough indexing against. Some have a one box where they attempt to answer your question with the first result. Others have refinement tools to help you drill down. But all of them have a robust set of results they return.

It’s those “little queries” that truly form our habits when it comes to choosing a default search engine. If you started with Google, you probably stayed there because it does a good job of delivering on those and you got used to the layout. But, to Patrick’s point, if you started with Live Search recently (even Patrick admits that Live Search at launch was not that good) you’d probably stick around because it does well against those “everyday” searches.

Putting his money where his mouth is, Patrick asked us to commit to Google-Free Fridays and if, after a few weeks, we didn’t think Live Search was every bit as good as Google he’d, well… he never told us what he’d do so, Patrick, if you’re reading this, leave a comment and let us know what’s at stake here.

I told Patrick we’d give it a shot but with a rebranded name and an open opportunity for search engines of all shapes and sizes to make the rotation. To be sure, we’d give Live Search the first crack at the spotlight though.

So, today, we’ll be using Live Search as our default engine and we invite all other Resolution Finders to do the same. We’ll report back next week with what we found and how it can be applied to search marketing programs. And we’ll also pick the next search engine in the rotation for the following Friday.

Happy SERFing!

12 comments:

Patrick Harris said...

Thx for the post Aaron...If your findings do not meet my hypostasized outcome, perhaps I’ll show up at the RM offices in the butterfly costume to get further insight into your methodology.

Cheers…Patrick

Aaron Goldman said...

It's a deal Patrick!

dgould said...

I made a conscious effort to go with Live Search throughout the weekend. Although I didn't see much of a difference in functionality doing web searches, I was a big fan of the image search functionality. Live Search enables the searcher to sort images by a number of different criteria that I found helpful. Live Search also allows the searcher to add images of interest to a "scratchpad" for later reference. Lastly, it has a sliding bar scale at the top right that allows the user to easily adjust image size to fit more or fewer images on one page.

With regards to relevance, i purposely mirrored a few of my searches on Google. Again, I found little difference in the web results, but for image results the story was not as promising for Live Search. the number of returned images was dramaticaly less than Google. Some might argue that fewer results means these results were more relevant and contained less clutter. i did not find that to be the case. on the contrary, I found that Live Search was leaving very relevant images out of the picture (no pun intended, of course).

In summary, I felt Live Search stacked up for web results. For image searches, I loved the functionality, but am suspect about the quality of the results returned.

CJeffCampbell said...

Wow, so I searched "Berco do Infante 2006 Reserve" (for my wine blog) and the sponsored results were: army.com, Cars.com, Stub Hub, & McGrath Lexus of Westmont. Army used the word "reserve", but the others couldn't have been more unrelated to my search or each other. Very poor/odd relevancy. Run of site ads?

http://search.live.com/results.aspx?q=Berco+do+Infante+2006+Reserve

CJeffCampbell said...

A few more comments to add:

-agree with DGould on superior image search

-Birds Eye view on live.com maps was also excellent. Google map's satellite view shows my condo, which was built 4+ years ago, as a vacant lot still. Live.com let's me show folks how large of a party our rooftop deck can hold.

-I didn't notice many blogs in the search results. In fact, for both this company blog as well as my personal blog traffic from live.com is way, way, way under 1% of the total. Searchers love opinion. However, I did notice, when searching my name (ah, the ego search) my StumbleUpon, Digg, and LinkedIn social media profiles show - interesting.

Aaron Goldman said...

Got a live demo last week from Irving Kwong, who runs the team responsible for developing Live Search. He pointed out some cool features that I was not aware of.

-"Phoenix traffic" pulls up real-time traffic

-Live Search news includes local news and includes video thumbnails that play on the SERP when you mouseover

-"flight from seattle to phoenix" integrates Farecast information on the SERP showing if the fare is expected to rise or fall

-Local search queries for restaurants have an opinion index which aggregates sentiment from reviews

-Live Search toolbar monitors searches on other engines and tries to draw you to Live Search with auto-population and blinking buttons (eg, when you search for "Canon 5d" on Google, the Live Search toolbar on your browser automatically puts that query in the box and then pulses the search and Cashback buttons)

-The new Live Search homepage includes images from obscure locations and allows you to click on various parts of the image to see obscure queries about that locaation. Trying to show you that search can be used to discover.

While these features are certainly useful and I applaud Irving and the team at MSFT for its continued innovation, there are some basics that are still missing from Live Search.

-A query for my address does not pull up a map on the SERP

-When searching a bunch of celeb names, only Britney returns image results

-A search for "Lazy Cloud Inn" (which is in Wisconsin) returned ads for hotels in Atlanta

-"Digital Sea Change" does not return my blog DigitalSeaChange.com (Google indexes it in the top spot)

-"SERF Live Search" does not return this blog post! (Google has it #1 along with an ad for Live Search)

-Lots of untargeted cashback ads. It's almost like ebay all over again bidding on every keyword ("Buy Aaron Goldman on ebay"). I searched "Drugstore.com promo codes" and it returned an ad for "promo codes" on cashback.

-"United Center" returns ads in top sponsored spot for TicketsNow.com and SourceTix.com and then shows the same ads again at the bottom of the page.

One overall observation is that (with the exception of the last query where the same ads were shown in multiple spots on the page) there are fewer ads on Live Search SERP's than Google. Now this can be viewed as both a good and bad thing. From a consumer POV, fewer ads means less clutter but it also means less relevant results to choose from (again, assuming the ads are relevant - see "Lazy Cloud Inn"). We all know though, that the reason there are fewer ads is because Live Search query market share is low so many advertisers don't run on it, opting to go thru Google and Yahoo instead.

Bottom line, Live Search has definitely closed the gap on Google. There are tons of queries I performed and didn't list here because the results were pretty much the same as the Big G. At the end of the day, though, being "as good" is not good enough to close the gap in market share. Looking forward to seeing what Irving and team do next. And, of course, will be keeping an eye on MSFT's continued distribtuion efforts because if a tree falls in the forest and no-one's there, it doesn't really make a sound.

Patrick - for the purposes of our bet, I'll concede. How about we make a follow up bet though? If your market share doesn't hit 20% by this time next year, will you dust off that butterly suit?!?

Matt Cutts said...

Aaron, just out of curiosity, how much your opinion came from using Live Search for a prolonged amount of time vs. getting an in-depth demo from a Microsoft person?

Aaron Goldman said...

Matt - I've actually made a concerted effort to use Live Search on a regular basis. Granted, "regular" for me means Search Engine Rotation Fridays whereas the heavy majority of my day-in and day-out searching is still via Google.

To your point (at least I think you were making a point and not just being genuinely curious), my POV was definitely skewed by the demo I got from the MSFT folks. It's unlikely I would've come across those features unless I happened to be checking traffic or planning a trip on a Friday.

That said, you have to admit MSFT has closed the gap over the past couple years in terms of index size and relevancy.

However, as noted in my previous comment, my non-demo usage showed some basics still missing from Live Search.

Bottom line, while it's great to have some fancy results for travel, shopping, and news queries, I'm not going to kick the Google habit anytime soon. Bells and whistles are no replacement for depth and consistency.

So keep doing your thang Mr. Cutts!

Matt Cutts said...

"you have to admit MSFT has closed the gap over the past couple years in terms of index size and relevancy."

That hasn't really been my experience. I agree with a lot of the searches that you posted where Google does better than Live/MSN. The metrics that I know of also support that Google is still much better than Live in terms of relevancy, index size, and other elements of the search engine secret sauce. :)

Aaron Goldman said...

Matt - I'm not saying they're close, just closer than they were a few years ago. C'mon, surely you can give them that?

Matt Cutts said...

Aaron, MSFT's search has certainly improved since their launch (was that 2004? Has it really been 4-5 years?). But Google hasn't remained standing still; we've improved our search in lots of ways since then too.

In my experience, the first few improvements are easier than getting quality wins once relevance is already high, so the rates of improvement are hard to compare.

Aaron Goldman said...

Fair enough, Matt. Thx for weighing in. Looking foward to more healthy competition and continued innovation.

 
Copyright © 2008 Resolution Media, Inc. All rights reserved.