So much for do no evil?

Why doesn’t this Google search for “feedster” return a link to www.feedster.com? Microsoft seems to have no problem linking to their competition. This is a little scary given Google’s ability to define reality. It’s now a routine conversational turn for me to say to friends, “Google it and you’ll find the Web page.” It’s too early to tell but I’d hate to see Google become the FOX News of the Web.
Update Feedster de-Googled. Scott says, “Google has pulled us quite thoroughly — you can’t even find out who’s linking to us.”
Update 2 Actually http://www.feedster.com doesn’t appear in MSN or Yahoo! searches either. Maybe it’s a problem with the robots.txt file?

3 thoughts on “So much for do no evil?”

  1. I feel like a lamer doing basically the same post in three different places (John Battelle, Scott Johnson, and now your blog), but just wanted to try to help explain what I think is causing this.

    I think I found the reason why the query [feedster] doesn’t return http://www.feedster.com at #1: it appears that Feedster is serving different content depending on the user agent. A normal user gets normal content, but when the user-agent contains “Googlebot/2.1,” Feedster serves up a 404. I verified this from the Googleplex and from my school account, but you can try it for yourself as well:

    # Fetch as a normal user
    % wget http://www.feedster.com/
    –21:39:51– http://www.feedster.com/
    => `index.html.8′
    Resolving http://www.feedster.com... done.
    Connecting to http://www.feedster.com[66.151.189.204]:80... connected.
    HTTP request sent, awaiting response… 200 OK
    Length: unspecified [text/html]

    [ ] 6,266 75.54K/s

    21:39:51 (75.54 KB/s) – `index.html.8′ saved [6266]

    # Now fetch with a Googlebot-like user agent
    % wget -U Googlebot/2.1 http://www.feedster.com/
    –21:40:02– http://www.feedster.com/
    => `index.html.9′
    Resolving http://www.feedster.com... done.
    Connecting to http://www.feedster.com[66.151.189.204]:80... connected.
    HTTP request sent, awaiting response… 404 Not Found
    21:40:02 ERROR 404: Not Found.

    Given that the issue appears to be on Feedster’s side in this case, if Feedster will start serving Google a normal page again instead of a 404, I would expect their root page to be back right as rain in a few days.

    Okay, I’m off to see read what Jeremy Zawodny is saying about televisions.. 🙂

    Like

  2. Sure enough, looks like the query [feedster] brings up the root page http://www.feedster.com just fine again on Google now. I’d be curious if anyone can say why Feedster was serving 404’s to search engine spiders but not to regular users. Anybody know?

    Like

Comments are closed.

%d bloggers like this: