Search engines & IDX Part VIII: Options for moving ahead

We've considered this issue in seven posts:

• Part I: What's the beef?
  
• Part II: Search engine scope and data sources
  
• Part III: Search engines indexing, audiences, and their objectives
  
• Part IV: "IDX index fishing"
  
• Part V: Is search engine indexing legal?
  
• Part VI: Purpose of IDX and broker expectations
  
• Part VII: Miscellaneous items
  

I promised that breaking this topic up would keep me from five-page blog posts, but I've lied. In this last marathon scheduled post on this topic, we'll look at some solutions folks have proposed and see whether they resolve the situation.

Interests to advance

A good solution to this problem should advance several interests, and I'll use these to evaluate the potential solutions below:

1. Listing Broker Expectations: Establishing reasonable expectations in listing brokers about how their listings may be used on the IDX sites of their competitors. This does not mean accepting brokers' current expectations, because in many cases, those expectations may result from lack of knowledge. Rather, it means telling the brokers, by rule, education, or a combination, what to expect. Any use beyond those expectations should be considered misuse.

2. Supporting Broker Web Efforts: Supporting efforts of brokers to be the 'go-to' sites on the Internet for consumers looking for real estate. This means giving brokers the tools to attract and retain consumers to and on broker sites. I think this is even more important than the first factor: after all, through education and clear rule-making, we can change listing broker expectations. But nothing MLSs do is going to alter the expectations of consumers on the web. We need to equip brokers to face the realities of the web.

3. Technology Tools: Exploiting technology that prevents obvious misuses of data at the source, if there are such tools. This means placing responsibility with the displaying broker (the one hosting the IDX site) to prevent obvious 'scraping' efforts, to the extent they can be distinguished from permissible uses.

4. Legal Tools: Giving listing brokers and MLSs legal tools necessary to go after those who misappropriate and misuse the MLS data. Once the data is out of the hands of the broker operating the IDX site, neither that broker nor MLS has physical means to prevent misuse of the data. So listing brokers and MLSs need other tools to go after 'the bad guys.'

Of course, the challenge will likely be balancing listing broker expectations and the efforts of site-hosting brokers to attract and retain consumers by all legal and ethical means. To balance those interests, I wish we had a little more info on these two points:

First, what are the types and frequencies of these searches on search engines? How often, for example, does a consumer go to Google and type: "Nancy Smith" if she's looking for a listing of Nancy's? How often does a consumer go to Google and type "123 Elm Street" when she has driven by a listing? If consumers only infrequently use these "long-tail" searches, permitting them may not be critical to the success of broker SEO efforts.

Second, if we were to talk through how IDX index fishing works with a group of listing brokers, would they be distressed? I've argued that listing brokers would be justified if they felt IDX index fishing frustrates their expectations about how IDX works. But no one has demonstrated that listing brokers as a group actually care. Am I wrong about that?

I don't list clarity as an interest, because clarity is essential - being clear about what is required, what is permitted, and what is forbidden. Rules that are not clear do not provide a good basis for conduct. As far as clarity is concerned, I don't think anyone as satisfactorily defined the distinction between legitimate indexing by web search engines, entities not in the real estate business that are indexing the whole web and that promptly pass consumers off to matching sites, from "indexing" that is really scraping by a real estate player. If we permit 'search engines' without clarifying what we mean by 'search engine,' anyone can scrape data from IDX sites all over the country, put it up on a national 'real estate search engine,' and attempt to monetize the resulting traffic by selling leads back, etc.

Options

So, what are our options? Well, NAR's current rule reads this way:

Participants must protect IDX information from misappropriation by employing reasonable efforts to monitor and prevent "scraping" or other unauthorized accessing, reproduction, or use of the MLS database. (Section 18.2.2 of NAR's model rules, NAR Handbook on Multiple Listing Policy, 2009 ed.)

Option A. No change – Allow each MLS to interpret the rules

If NAR does not change the text of Section 18.2.2 of its model rules, I expect MLSs will continue to interpret the section as they do now, with some holding that web search engines are 'scrapers' and other that they are not.

The current rule has a clarity problem: The first sentence imposes on the broker a duty to prevent misconduct that is beyond the sphere of the broker's control ("monitor and prevent ... unauthorized ... reproduction, or use"). A broker can be asked to prevent scraping of her site, using commercially available tools to monitor access to the web site to detect scraping and terminate access from IP addresses that engage in it. (This may not ultimately prove successful, but it may at least make scraping more difficult.) But she can hardly be expected to police uses by a third party unless the way the third party 'scrapes' the site gives the broker notice of some kind of problem.

It's also unclear in that we don't know whether web search engines are making authorized use or unauthorized use of the MLS data.

It's unlikely NAR will allow local MLSs to make these decisions. A key reason is that brokers who function in multiple markets will not like it if two adjacent MLSs take opposite views. The arguments those brokers make about the difficulties this would pose for them might be overstated, but NAR will hear the arguments nevertheless. NAR has attempted over the last several years to make IDX rules more uniform, and I expect they will respond on this issue in kind.

Option B. All MLSs adopt the MIBOR approach

NAR could require all MLSs to adopt the MIBOR approach, prohibiting indexing by search engines. Then I'd recommend changing section 18.2.2 of the model rules to read:

Participants must make commercially reasonable efforts to prevent the gathering of MLS data from their IDX sites by automated means, including techniques commonly described as "scraping," "spidering," or "web-crawling." For purposes of this Section, indexing by search engines constitutes "web-crawling."

By the way, I don't think this approach would prevent listing brokers from doing IDX index fishing with their own listings. Generally, NAR and MLS policies do not restrict how a listing broker displays her own listings. So, in this case, even if MLS says "you can't allow search engines to index IDX listings," a broker could permit her own listings to indexed on her own web site.

Thus, there are many who would claim the big listing brokerages were driving this approach. I think that it's reasonable for a listing broker to say, however, "If someone searches for my listing agent or the address of one of my listings on Google, the result should be a link to my site or to a neutral site that I have authorized to display my listings, not to my competitor." The listing broker who agreed to take part in IDX to allow other brokers to advertise her listings may never have contemplated that her listings would be used as search engine bait.

As for our interests, this approach does a good job of setting listing broker expectations because it's clear what is permitted and what is not. As this approach does not require making a distinction between good scraping and bad scraping, it does not require nuances in interpretation. There are also technology tools that permit web sites to prevent almost all spidering, web-crawling, etc. (rather than requiring the server to attempt to distinguish between good bots and bad bots), so it supports our third interest. As for legal tools, this approach provides none, but adopting Option E in conjunction with it could change that.

This approach fails to support broker web efforts, because non-broker sites will be able to take advantage of web search engines, but most broker sites will not. This is only one of five interests weighing against this approach, but frankly, I think it's critically important. Do we really want to say that brokers with IDX sites cannot use site development techniques that are common across many industries and that are otherwise legal and ethical? Do we want to permit aggregators to use tools that we do not allow to brokers?

Option C. NAR policy committee approach from May 2009 (or modified form)

NAR could adopt the policy proposal it considered in May, which essentially permits all indexing by search engines. This is how that change would have left Section 18.2.2 of the model rules:

Participants must protect IDX information from unauthorized uses. This requirement does not prohibit indexing of IDX sites by search engines.

This also suffers from clarity problems similar to the ones in the current policy (Option A). Here's language I'd propose instead:

Participants must make commercially reasonable efforts to prevent the gathering of MLS data from their IDX sites by automated means, including techniques commonly described as "scraping," "spidering," or "web-crawling," except when those techniques are used by web search engines. A "web search engine" is [define here].

(Note that I have not defined what a "web search engine" is or what data uses typify a "web search engine." This is not a trivial problem. If you have a sense of how to define it, post a comment.)

It is essential that the policy language define "search engine." As we concluded previously, any database that a consumer can search that displays the results online is a "search engine." Trulia, Google, and Realtor.com are search engines. In fact, almost any site likely to misappropriate IDX data is likely to do so by presenting it in the form of a "search engine." When evaluating this approach, I'll assume it comes with a good definition of "web search engine."

This approach should establish clear listing broker expectations (though I think we'd have to educate listing brokers about how this works, what they might see, and what their own options are for using the same techniques). It also supports broker web efforts, essentially allowing IDX to compete with the likes of Realtor.com, etc. It does not really provide any legal tools for stopping bad actors, but adopting Option E in conjunction with it would help.

This approach is a mixed bag on the technology front. The broker hosting an IDX site might be able to detect that someone is scraping the site, and as long as the scraper is not posing as a web search engine, we could probably expect the broker operating the site to take steps to prevent the scraping. But it would be difficult for the broker know if a malicious scraper is posing as a legitimate web search engine; in other words, the displaying broker's "commercially reasonable" efforts are unlikely to prevent a determined scraper.

This approach will not be without its detractors: I think listing brokers do not expect that searching "Nancy Smith," a listing agent with ABC Realty, on Google, will result in a match to one of Nancy's listings being displayed on XYZ Realty's IDX site. I know there are those who discount this listing broker concern as symptom of a failure to compete on the web. But IDX after all, is a grant of permission by listing brokers for other brokers to display their listings. They are entitled to have a say about how the listings will be displayed. If you want to do funky stuff with other brokers' listings, you can do it behind a password on your VOW (though web search engines should not be able to index VOWs). Might big listing brokers pull out of IDX if NAR adopts this option? I'll save that question for a post on the strategic interplay between IDX and VOWs.

On the other hand, if the information I've shared in previous posts (and comments of others associated with them) is correct, listing brokers can readily adopt the techniques of "IDX index fishing"; and widespread adoption of these techniques will make them less effective, and presumably, less distressing to listing brokers. Perhaps the way to address listing brokers' legitimate concerns is to educate them about how IDX index fishing works, and about how they can make use of it themselves.

Option D. Permit indexing, but limit fields that can be indexed

NAR could adopt an approach that permits indexing generally, but prohibits it on a few key fields to address the listing broker concerns about the example I gave in Section C. The rule might look like this:

Participants must take commercially reasonable steps to prevent the gathering of MLS data from their IDX sites by automated means, including techniques commonly described as "scraping," "spidering," or "web-crawling," except when those techniques are used by web search engines. A "web search engine" is [define here]. If a Participant permits or encourages web search engines to index the Participant's IDX site, the Participant must not expose the following fields in the listing display provided to web search engines: name of listing broker, name of listing agent (or non-principal broker), street address of listing [Others?].

Generally, this approach has the same virtues and vices as Option C, except this one does address a narrow concern of listing brokers seen in the example I gave in Section C (search for listing agent's name shows link to agent's listing on a different broker's web site).

This approach does create a sort of paradox, though. Most MLSs have rules requiring that an IDX display include the listing broker's name. It seems odd to require an IDX site to withhold the listing broker's name when displaying that broker's listing to a search engine. Essentially, I think this rule would require the broker to provide a different display to search engines than it provides to consumers. That makes me uneasy, though I can't say exactly why.

Option D.5? One possibility arises from the fact that many MLSs do not require that listing agent name be displayed on IDX (only listing broker name). By prohibiting display of listing agent name in IDX, MLSs would prevent the problem in the "Nancy Smith" example above, but without having separate display rules for search engines and consumers. This option probably does not work in states that require the listing agent to be identified on advertising.

Option E. Control downstream uses with terms of use

Mike Wurzer described his notion that we should impose a "terms of use" requirement on users of IDX sites. The NAR model rules (see model rules Section 18, pp. 118-21 of the NAR Handbook on Multiple Listing Policy, 2009 ed.) for IDX do not require that IDX sites display terms of use. We have urged MLSs for years to adopt such requirements (for reasons unrelated to the current dispute).

The law about online contracts has evolved. Most courts are likely to find an enforceable contract where the visitor to a web site clicks on a "SEARCH" button that has a link to terms of use next to it with text that says "By clicking 'SEARCH' I am agreeing to this site's terms of use."

Our MLS clients that have adopted requirements for TOUs to appear on IDX sites provide the actual text of the terms of use, or at least key portions of it, to their brokers. One common TOU provision says that the MLS is a third-party beneficiary of the agreement the TOU creates. That means that the MLS can sue the consumer/web site user if he/she/it breaches the TOU and misuses the data. Other provisions embody some language from the NAR rules about "personal, noncommercial" use.

No matter which option NAR adopts for model rule 18.2.2, I'd recommend that it also adopt an optional IDX rule that allows MLSs to impose terms of use on visitors to IDX sites. The advantage of this approach is that it allows the MLS to police the downstream uses of the data of which it becomes aware. This complements any efforts the displaying broker may make server-side to prevent scraping.

(BTW, though this suggestion comes from a post by Mike, I don't support a suggestion he made in the same post that it would be better to limit the fields available in IDX to alter the impact of search engine indexing. I'll take that up in a post on strategic interplay of IDX and VOWs in coming weeks.)

What I would probably do...

So, if NAR said to me, "We're going to take action on this, what do you recommend?", I'd probably say this:

• Define what we mean by 'web search engine,' identify the benign uses they make of listing data, and incorporate those descriptions into the rules.
  
• Say that broker IDX sites may allow and even encourage indexing by web search engines.
  
• If brokers are particularly miffed by the "Nancy Smith" example, MLSs can prohibit display of listing agent in IDX (as long as state law does not require it).
  
• Educate all brokers about how site indexing works and about technology options to allow them to take advantage of it.
  
• Allow MLSs to adopt a rule requiring IDX sites to display terms of use prominently on the site (but giving brokers a few months' grace period to implement them).
  
• Develop a good model TOU and invite the MLSs to promulgate it to brokers; two key terms would make MLS a third-party beneficiary and would allow 'web search engine' use but not any other commercial use of the listing data.
  

I'd chose permitting indexing over prohibiting it, because I think we have to let brokers have the tools they need to compete on the real web. Listing brokers who don't like the results can employ the same tools. I don't like Option D, but as I say, I can't quite put my finger on what there is about it that makes me uneasy.

I would stress the need to define exactly what constitutes benign "search engine indexing."

I would also stress the need to put together some education to help MLSs and brokers. NAR has CRT, which is a great resource, but it still tends to make knee-jerk decisions and then allow local MLSs to have to deal with the implications. It's down-right idiotic if NAR does not produce material that explains how this works in layman's terms.

As for the model terms of use, I'd suggest that NAR consult with attorneys of some of the MLSs about this, rather than basing the model on NAR's purely theoretical understanding of the issues.

Your ideas?

-Brian

Current legal issues with VOWs? (Potential CMLS topic 1)

(Note: Shelley Specchio is CEO of the Northern Nevada Regional MLS, Inc., a host of the CMLS Conference in Lake Tahoe, September 30 – October 2. She and I have been discussing topics for the legal panel there. Shelley wants input and feedback from those likely to attend: Which legal topics are of greatest interest and what aspects of them are most important for MLSs? I agreed to do a series of blog posts on some of the candidate topics, cross-posting links to them in other forums and asking folks for their input. This is the first. If you have other topics to suggest, email me or comment on any of these posts.)

NAR has received its first written complaint from a broker regarding an MLS's implementation of the VOW policy. My law firm represents the MLS about which the complaint was made; and we believe the matter is under control and that the MLS has complied with the policy at all times.

Nevertheless, we expect a few more problems may crop up with MLSs and the VOW policy before the CMLS meetings, so we will be prepared to discuss them then.

My suspicion is that problems will develop for a number of reasons.

"Confidential" fields and statuses

There still seems to be quite a bit of confusion about what it means for an MLS to designate a field or status as confidential under the VOW policy. I've posted on the issue here and here. Some MLSs are assuming that anything that does not appear on a "customer handout" format in the MLS is confidential. Others have adopted NAR's model rule Section 19.15 (or one of the other 'parity options'), but have not adopted parallel language elsewhere in their rules restricting the disclosure of those fields/statuses in all other media, including orally, which is what the VOW policy requires.

Technical implementation

To implement the VOW policy, MLSs really probably need to have four fields on the listing database to capture the seller's options under the VOW policy (relating to listing and address display on the Internet and relating to automated valuation and third-party commentary). Unfortunately, some MLSs have given confusing names to these fields, and listing input staff and agents doing input at brokerage firms may not understand their implications or even know about the new VOW rules. MLS staff also need to be sure that the seller selection for "no-display-on-Internet" prevents applicable listings from being displayed on Realtor.com, aggregators sites, and in IDX if the MLS will be excluding such listings from their VOW data feeds, which is what the VOW policy requires.

Listing brokers not understanding the new policies

In the near term, most brokers likely will not operate VOWs. Their agents nevertheless need to be aware of the policy so they can advise sellers about their options and provide a heads-up about how the seller's listing will be treated on other brokers' web sites. We've given "Virtual Brokerage Now" presentations for a number of MLSs to help with this education. Though the programs are well-attended and well-received, we really are just scratching the surface of the pool of folks who need to be educated.

Forms problems

I did not anticipate this, but forms have been a problem. Many state associations and some local associations and MLSs develop their own listing contract forms. Unfortunately, a number have not properly understood the policy and have created forms that do not accurately represent the seller's options.

The seller has four options:

1. withhold her listing entirely from the Internet (including VOWs, IDX, Realtor.com, and aggregator sites);
   
2. withhold her listing's address entirely from the Internet;
   
3. prohibit displays of automated valuations of her property adjacent to it on VOWs (this option only affects VOWs, though I expect the IDX policy will be changed to cover this at NAR's meetings in November);
   
4. prohibit displays of third-party commentary about her property adjacent to it on VOWs (again, I expect this will apply to IDX soon).
   

(One issue is whether these options should appear in the listing agreement, where the seller normally authorizes marketing in broad terms, or if they should be in a separate form. We think that both can work, but we have recommended the separate form for reasons I can explain in another post if anyone expresses interest.)

Among the problems with forms that we have seen, for example:

• One association's form gives the seller options (i), (ii), and (iv), above, but inexplicably omits option (iii).
  
• One association's form informs the seller with regard to options (iii) and (iv) that she can opt out of displays of automated valuations and third-party commentary "on the Internet." But those seller's options currently only extend to VOWs, not to IDX and certainly not to the many other web sites her listing will likely end up.
  
• One association's form tells the seller she can opt out of "blogging." I presume this is meant to refer to third-party commentary. But the term "blogging" is both under-inclusive (because third-party commentary can take forms other than blogging) and over-inclusive (because blogging by the broker displaying the listing is permitted despite the seller's request, if the blog post expresses the displaying broker's professional judgment).
  

There are many others. Unfortunately, many listing agents do not understand the policy well enough to explain the options even when the forms are correct; the result is that sellers' expectations are likely to be frustrated – never a good outcome for the listing broker.

What other parts of the VOW policy do you think are likely to create problems? Have you had any issues with it you would like to see discussed?

(Tomorrow, I expect to do the final planned post regarding IDX sites being indexed by web search engines.)

-Brian

Search engines & IDX Part VII: Miscellaneous items

This posts covers a couple miscellaneous items I wanted to get out before doing the final post in this series.

"Static pages" and "RESTful development"

Mike Wurzer pointed out that my post on how IDX index fishing works is technically inaccurate. I asserted there that IDX index fishing requires the creation on the IDX site of static pages showing all the listing information. He explained that current web development methods are focused on creating sites that are easily indexed, and that they tend to include links (or what I have previously called 'sitemaps') that link to dynamic content. In this way, they get the benefits of having static, indexed pages, as I described them in Part IV. He described this as "REST" (which stands for "representational state transfer"). I did a little (admittedly, a very little) reading about "RESTful" development on Wikipedia. I also visited a number of web sites that deliver database-driven content; RESTful techniques appear to be in wide use throughout the web.

Based on these facts, I'm prepared to revise the implications of my earlier comments. There I said that "IDX sites are not naturally prone to be indexed by web search engines" and that "some brokers with IDX sites are intentionally using the listings of other brokers in IDX to fish for indexing." The implication is that brokers using these techniques are somehow departing from the norm. In fact, after Mike's helpful comments and a little research of my own, this appears to be the norm in the design of sites driven by database content. I'd be curious if anyone can offer evidence to the contrary.

Users posing as Google robots

Victor Lund made a comment on an earlier post noting a blog post that purports to show how anyone can pose as Google and thus index a site, even if it's behind a registration (which could have impacts on VOWs, if true). I reviewed the post, and it appears that the site has to allow this conduct. In other words, an IDX site that required registration before making certain content available, or a VOW, which is required to make visitors register and login before showing them VOW content, would prevent the technique described there. (The comments to that blog post clarify it.)

In any event, with IDX index fishing, we're talking about broker sites that want to be indexed, not ones that are avoiding it. Under the VOW policies, a VOW-operator may not allow the VOW to be indexed by web search engines.

More IDX index fishing may make it ineffective

Mike Wurzer pointed out on an earlier post that permitting IDX index fishing may actually lead to the method becoming less effective. This, as it turns out, may be a very important point.

Web search engines like Google apparently have "duplicate content filters" – these are designed to screen out web pages that contain content very similar or identical to pages on other web sites. The web search engines are trying to prevent what some call 'search engine spam' – dressing up the same content on multiple sites in order to increase search engine rankings or traffic.

I found several easy-to-understand summaries of duplicate content filters. (Here's one, though I don't claim it's accurate.)

The upshot: If IDX index fishing techniques appear on every broker IDX site, or even most of them, they may find that all their pages are being filtered as duplicate content by the likes of Google. Consequently, if IDX index fishing becomes widespread, it may also become largely ineffective. (Of course, I expect clever web designers and SEO contractors will look for ways around this 'problem.')

On the other hand, according to an article on the topic on Google's Webmasters/Site owner help, it looks as though Google may try to 'pick a winner' among the many sites that have duplicate content. In that event, I'd be worried that one broker with clever SEO might effectively monopolize the top spot. Such an advantage is still likely to be temporary, though, as other brokers, SEO experts, and aggregators like Realtor.com will be spending time and money to overcome it.

One more copyright issue

In response to my post about the legality of web search engines indexing web sites, Rob Hahn posted a comment to which I want to respond. Here is an excerpt:

Something I'm curious about... is the difference between the Kelly case... and the case of listings. A listing, after all, is more or less a compilation of facts about a property. It isn't a creative work, like an artistic photograph. Furthermore, what of the relationship between the seller and the listing agent? Supposedly, the listing agent is a fiduciary of the seller, whose home is the ultimate property at issue. Wouldn't the seller hold the ultimate IP to descriptions, likeness, facts of the house, and any grant of license to the listing broker is dependent on the seller? It's a tricky area, but how would you go about unravelling the original and base IP at issue -- that of the seller? How does that impact the whole IDX/Search issue? I rather think there's an impact here but maybe I'm overthinking it.

It's important to distinguish the rights of the seller under the listing contract, as the owner of the property and perhaps principal in an agency relationship with the listing broker, from the rights of the listing broker/agent in the listing record
under copyright law as author of creative content. Copyrights in the listing photographs and textual descriptions of the property belong to the human being who created them – usually, but certainly not always, the listing agent. (There are some exceptions – probably not important here.) And photos and descriptive text of the kind common in MLSs are both creative works from the perspective of copyright law. (So too is the "compilation" of the facts in the MLS database, but that's a topic for another post.)

Generally, then, the seller would have intellectual property rights in the listing record only if the listing contract (or some other writing) transfers them to the seller from the listing broker/agent – or, of course, if the seller had created the content in the first place.

We could spend a long series of posts on copyright issues in the industry, but I think readership would drop dramatically in that event... Maybe I'll see if I can summarize in a single post somewhere down the road.

-Brian

Search engines & IDX Part VI: Purpose of IDX and broker expectations

We're almost to the end of this line, I think. I took a break for a couple weeks to think more about this and to talk to some thought leaders (and also to do a little of the work for which I get paid ;-). In this post, I'll spend a little time on the history of IDX and what I think most brokers think its purpose is. I'll argue that IDX index fishing, though not illegal or immoral, 'breaks' the expectations of most listing brokers regarding IDX, but that we probably should find a way to permit it. In the next post, I'll talk about a couple technical and legal details. And in my final planned post (number VIII!) I'll discuss the specific proposals of which I'm aware and wrap it up.

A little history

I sometimes get described as "the father of IDX." I think that's an unfair characterization. Of course, anyone who's known me long knows that I am head-over-heels-in-love with the idea that I believe underlies IDX: "The best source for listing information on the web should be the web site of a broker participating in MLS." IDX facilitates this by ensuring that a broker with an IDX site can display nearly all the active listings in the MLS to visiting consumers. In almost all MLS markets (with some quirky and distressing exceptions), that means the broker's site can be unsurpassed as a source for listing information.

Back in 1999, when I managed the RMLS in Minneapolis/St. Paul, we looked at the model adopted at the broker-owned MLS in Seattle, Northwest MLS. They called their program "Broker Control." RMLS considered whether it should adopt the same approach. While I was on a three-month sabbatical, two key members of my staff, Rachel Wiest and Rebecca Younger, worked tirelessly with a small group of RMLS's leaders, most notably Phil Olson (a Coldwell Banker broker who sadly died just a couple years later at a tragically young age) and Henry Brandis (a leader at Edina Realty). Late that year, after I returned from sabbatical, we tweaked the rules to address the concerns of a big listing broker in our market.

As soon as RMLS launched IDX (we called it "Broker Reciprocity"), I started crowing about it on listservs (like those maintained by Internet Crusade) and via other means. I spent the balance of 2000 and 2001 traveling all over the country explaining IDX to large groups of brokers and MLS execs. In 2001, on behalf of RMLS, I wrote the IDX Implementation Guide for NAR, which NAR published in August of that year, just a few months before the implementation deadline of January 1, 2002. I also moderated a listserv at for the Internet Crusade on IDX or broker reciprocity issues. Over the years, I may have talked with more brokers and MLSs about the 'why' and 'how' of IDX than anyone else – I've heard a lot about their expectations.

The purpose of IDX

There is no definitive statement of the purpose of IDX. NAR did not adopt one with the IDX policy in 2000. The NAR IDX Implementation Guide did not articulate one in 2001. Even RMLS did not adopt an explanation for why it had done IDX. But the message of the leaders at Northwest MLS and at RMLS was very consistent:

The purpose of IDX is to ensure that brokers are unsurpassed as a source for real estate listing information on the web.

I looked back at my communications and presentations from the 2000-2001 era. Back then, we focused on IDX as a tool for brokers to make their sites 'sticky,' to keep the consumers on their sites once they arrived. We expected brokers would get the consumers to their sites using traditional marketing; brokers were already spending heftily on it, and adding a web address to display advertising would not increase their costs. That angle also helped to persuade large brokers, who held most of the listings, that it was cool sharing their listings in IDX with smaller competitors. Yes, the large broker and small broker sites would have all the listings, but the large broker's greater marketing budget assured more traffic per capita (with the capitae here being those of agents). (It turns out we were wrong about that, too. Some small and medium brokers get much more traffic per agent to their sites than even very successful large-broker sites.)

Until a consultant showed me IDX index fishing a couple years ago, I had never considered the question of listings serving as 'bait' to get consumers to broker sites from search engines. Of course, they were always bait for the big listing aggregators in principle; I just had blinders on regarding their utility for SEO on broker sites.

Expectations thwarted, but maybe they need to change

I'm spending this time on history and the way things were to make a point: I expect many brokers in IDX share my antiquated views about how IDX ought to work, and that those expectations shaped the brokers' strategies to participate in IDX. Now that brokers have built web strategies around IDX, they cannot respond to expectation-breaking uses of the IDX data just by pulling out of IDX, as some have suggested. Preventing other brokers using your listings in IDX means giving up your own IDX. I can't imagine a broker doing that.

Thus, while I don't think IDX index fishing is illegal or immoral, I do think it departs from the expectations of brokers contributing their listings to IDX, particularly in that a consumer using a web search engine to search for the address or listing agent name of one broker's listing may get a search result showing another broker's site at the top. Listing brokers are right to be distressed about this. Brokers who get inquiries on their web sites about other brokers' listing are terrible at replying, resulting in a disservice to seller, buyer, and listing broker. (WAVGroup published research to this effect in "The Consumer Online Real Estate Search Experience" in March; Victor, I could not find a link....)

Practically, though, it may not be reasonable to ask brokers to hamstring their web-sites when it comes to competing with the likes of Realtor.com and other aggregators, many of whom are engaged in index fishing themselves. Thus, I think we need to find a way to make it possible for brokers' IDX sites to be indexed by legitimate web search engines. That also means educating all brokers in IDX about what their new expectations should be.

We'll talk specifics in post VIII on this topic, but we have a few little items to address in post VII first.

-Brian

Search engines & IDX (note regarding renaming)

For the four or five of you actually following the lengthy discussion on search engine idexing of IDX sites, I wanted to give you a heads up that I have renamed the posts on that topic to shorten the names (removing the academic colons, collapsing the two levels of title into one, etc.), with the goal of making the posts easier to find in the post history for MLSTesseract.

Sorry if this causes any confusion.
-Brian