Comments on MLS Tesseract: Search engines & IDX Part V: Is search engine inde...

mwurzer nailed it: IDX is all about putting li...

2009-07-29T23:01:41.100-05:00
mwurzer nailed it:

IDX is all about putting listings on the web through competing broker sites. If you're putting listings on the web, that means playing with search engines. Period. The policy is just fine as it is. Brokers can opt out if they want, but complaining about indexing when the purpose of IDX is to put the listings on the web is wrong.

Exactly.

Brian - I read this several times and sincerely a...

2009-07-19T00:28:42.387-05:00
Brian -

I read this several times and sincerely appreciate your insight. I too, would like to hear from more brokers and boards.

The whole topic is of great interest to me, not only because I was thrown into the fire storm; also, because I see this as a huge issue facing agents and brokers.

I have a few thoughts here:

If a broker is not aware he is

I found this article today that gives step by step...

2009-07-07T13:57:11.571-05:00
I found this article today that gives step by step directions to pretend to be a google bot - hinting that anyone can fake out a website and scrape data

http://www.addictivetips.com/internet-tips/access-any-website-or-forum-without-registering/

Brian - I thought the series is wonderful. Thanks...

2009-07-06T10:52:41.106-05:00
Brian - I thought the series is wonderful. Thanks for putting in the work.

Something I'm curious about -- and you as the IP attorney would be the go-to guy on -- is the difference between the Kelly case (which raises somewhat more traditional IP issues) and the case of listings.

A listing, after all, is more or less a compilation of facts about a property. It isn't

@Fred: You're right, it's really unlikely ...

2009-07-01T10:49:01.961-05:00
@Fred: You're right, it's really unlikely anyone is going to sue Google (or ther web search engines) in this situation. But my reason for doing the legal analysis was to show that what Google is doing is probably not illegal. Things might be different though if aggregator search engines like Trulia or Zillow stared indexing IDX sites, based upon the analysis I provided. In that case, I

I read it all! Thanks. Not dry. I am glad you bro...

2009-07-01T10:43:20.585-05:00
I read it all! Thanks. Not dry.

I am glad you brought up the Robots.txt defense. One might question whether the courts would allow defense that the Copyright holder should have Opted-out of indexing.

Lets assume there was no sitemap and no robots.txt.

Just like with spammers, the copyright holder can say "sure I didn't opt out with robots.txt, but when

I look forward to your next post, Brian, and I'...

2009-06-30T15:41:04.044-05:00
I look forward to your next post, Brian, and I'll foreshadow my comments:

Whether express or not, the purpose of IDX is and has been for a long time to allow competing brokers to put listings on the web. I also suggest that "on the web" means links and links mean search engines. Anything less than "on the web" means off the web. Also, as I've said already

I'm glad at least the two of your are reading....

2009-06-30T15:36:40.010-05:00
I'm glad at least the two of your are reading. I hope some brokers and MLS manager-types will chime in, too. I know some of them are reading...

Brian - you've nailed it - once we agree on th...

2009-06-30T15:35:07.832-05:00
Brian - you've nailed it - once we agree on the purpose and change the rules (that's IF we change the rules) then we can expect that broker site owners to comply, and search engines to follow rules such as robots.txt/noindex. It will just be important to make sure that somehow broker sites are not disadvantaged to others in some way. I don't believe purpose and policy discussion will

@Mike and Matt: Your comments do a great job of fo...

2009-06-30T15:09:27.951-05:00
@Mike and Matt: Your comments do a great job of foreshadowing my next post, which is focused on the strategic purposes of IDX. In short Mike, I think Matt is right to leave the issue "open" as you say, because the industry has never adopted a statement as to the strategic purpose for IDX. I think all of us have ideas about what it is/might be -- but the brokers whose listings appear

Matt, though we might agree on the basics, I'm...

2009-06-30T14:55:14.909-05:00
Matt, though we might agree on the basics, I'm stating my opinion that the core purpose of the IDX policy contradicts requiring a no-index tag, and you're leaving the question open. Brokers have a choice now through the opt-out, and that should be the mode of deciding this issue, not a blanket restatement of the IDX policy to require a no-index tag.

Michael - I think you've more or less summed u...

2009-06-30T12:12:09.105-05:00
Michael - I think you've more or less summed up my position - "If you're putting listings on the web, that means playing with search engines." We can put restrictions on search engines, but that means putting explicit restrictions on either what is put on the web via IDX or segregating it in some way so a policy / robots.txt file would apply. I'm not saying here that we

Issues of misappropriation always are fact specifi...

2009-06-30T09:07:13.207-05:00
Issues of misappropriation always are fact specific, and, yes, the facts can change, which also would change the conclusion. That doesn't mean, however, that we need to reach the conclusion before the facts change and shoot a shotgun to try to kill a mosquito. You'll most likely miss the mosquito and destroy what you want to preserve.

Though I disagree with the both the

@Mike: I'll accept your proposition to the ext...

2009-06-30T08:51:43.065-05:00
@Mike: I'll accept your proposition to the extent that it talks about Trulia and Google as they stand today, and I think I agree with it in principle. But that does not really help us draw the line from a policy standpoint. In other words, what if Google becomes a little less "conduitty" and Trulia becomes a little less "destinationy"? At what point does either cross the

Brian asked: "If a web search engine indexes ...

2009-06-30T08:30:35.610-05:00
Brian asked: "If a web search engine indexes the contents of an IDX search engine, is it transforming the content? Isn't it just copying the whole thing, possibly slightly transforming it, and then replicating the utility of the original? Don't three of the four factors now go against the defendant/web search engine?"

My answer is from your earlier post, which

Matt: I think what you are suggesting is that by p...

2009-06-30T07:38:44.649-05:00
Matt: I think what you are suggesting is that by permitting their listings in the IDX program, listing brokers are permitting them to be displayed on other brokers' IDX sites, and BY IMPLICATION, permitting them to be indexed on search engines because they are appearing on other brokers' sites.

I agree with you if, after acquiring an awareness of how these things work (e.g., by

Brian, I would go a bit further than your statemen...

2009-06-30T07:09:33.030-05:00
Brian, I would go a bit further than your statement that "By creating static pages of listings and detailed sitemaps...the site owner cannot complain when the indexing actually happens." - I believe that by placing information on the Internet (not just on a sitemap) without a restrictive robots.txt file one is implicitly allowing a search engine to index the information. Since brokers