• Project Manager - Trading...
• Head of IT Operations -...
• ICT Manager
• IT MANAGED SERVICES - BDM -...
• Technical Lead

The transformation of search, part 2

Bill French, Information Age

11/02/2005 13:52:17

I've long believed that the ultimate search technology is one you don't explicitly run; it works on your behalf in the background. Imagine turning the search paradigm on its head - instead of us finding stuff, why not stuff finding us? Pushing this idea to the extreme, our applications would understand what we are working on and automatically provide us with exactly the information that we need in every specific context. In that scenario, we would never need to search for stuff because the right stuff would find us and it would do so at the time and in a context that we appreciate.

This is the direction we're headed. We have a long way to go, but technologies like topic maps, smart tags, research services, and yes, RSS, are all bringing us closer. It's only a matter of time before information will be smart enough to hunt us down and make itself available to us when we need it most. Indeed a dream-like scenario, but there are already examples that this idea works.

RSS is a disruptive force. It is changing the way people connect with information and some would say, the way information connects with us. RSS lets you declare interest in specific subjects - encapsulated as channels - then sit back and watch new information auto-magically start finding you. Sure, you still need to go look in your newsreader to see the stuff that's coming to you, but your newsreader organizes information into channels that you are specifically interested in, greatly improving the signal-to-noise ratio.

And, the current generation of newsreaders is just the beginning. RSS is an XML standard information interchange technology. RSS (the technology) will continue to recede behind the scenes and become an implementation detail underlying many applications just as HTTP, ODBC, and countless other technologies have done before.

To get a sense of how RSS is impacting search consider that both Yahoo! and MSN Search now offer persistent search features through RSS. The idea is simple - get search results in the form of RSS instead of HTML so that a machine can watch it for you and report instances (search hits) that it hasn't seen before. This is a simple idea that is extremely powerful because it streamlines search in a way that it more productive for users; they don't have to visit the search engine to see if anything new has appeared in the search index.

Knowing what's new and skipping over stuff you've already seen is the holy grail of productivity. Integrating this benefit in all that you search for makes it an enticing model that enterprises should adopt as soon as possible.

Some would say that AI (artificial intelligence) has been a complete failure. Many people perceive that the basic premise of AI (machines that learn and think for themselves) has failed to produce any tangible evidence of progress in support of every-day solutions.

However, if you were to take a TiVo back in time a mere 30 years, consumers would be astonished at its intelligence; they would say that it is artificially intelligent and the AI research community would get the accolades they deserve, but never seem get to without the benefit of a time-transition perspective.

TiVo has a feature that is enabled by default when it is first installed - it watches your television-recording behavior and attempts to mimic it by recording programs that reflect your interests. I understand how it works, but I still think it's pretty magical; some would call it artificially intelligent. In my view, it's a form of search that reverses the paradigm; it provides a way for television programs to find me instead of me spending time looking for interesting television content to record and watch.

Indeed, TiVo is a form of RSS for television content, but the most important aspect (and worthy of deeper understanding) is how this works; simply stated, it's the meta-data. In the TiVo information service, programs exist within a taxonomy which is to say they have lots of meta-data to use for decision-making purposes. The meta-data makes it possible for TiVo to act on behalf of each customer; picking programs that you'd [probably] like to view - enabling programs to "find you".

It's clear that the future of search hinges on discretely tagged information. The trend to add (and rely) on meta-data for search purposes is everywhere. Consider the recent addition of the NoFollow attribute for links. This is a new attribute that is recognized by Google and when included in a Web link, Google will not follow the link for purposes of impacting page rank or developing a relationship between pages.

There are many use cases where NoFollow might be important, however, the more important aspect of this attribute is that it represents the beginning of a very powerful idea called typed associations; a way to classify links and allow us to create a deeper understanding of how information relates to other information objects.

For the purposes of determining relatedness, it's not good enough that a page links to another page - we need more discrete mechanisms for determining how important (or unimportant) a link may be. NoFollow is simply one small step toward the typing of links. The embrace of NoFollow allows us to shape (ever so slightly) what we mean by relatedness. I predict many more shades of grey will emerge (at the link level) to define how information relates to other information. The net effect will be an ever-increasing improvement in discoverability.

When we started MyST Technology Partners, we frequently made the comment that important information should find us instead of us always searching for it. The first overt instance of that capability came when we implemented SmartPoints - additional search results that indicate who is likely to know the most about a topic of interest and what channels are likely to hold more information about that topic. This is pretty compelling when you first start playing with it (the free public site provides this capability). The nature of this capability is more about clustering than search itself - the idea that we can also know what's close in terms of relatedness is extremely useful.

My hope is that as we continue to build new information systems that seem instinctively aware of activities surrounding you and your job or your personal interests, the system will naturally invoke behaviors that essentially say: "and BTW, I retrieved this information because you'll be needing it."