Understanding Search: Seven Critical Questions (and answers)
The Cast
Jim Baum, CEO, Endeca
David Bercovich, Product Marketing Manager, Google
Andy Feit, Senior Vice President/Intelligent Content Services Business Unit, Verity
Steve Kusmer, CEO, Atomz
David Luzier, Director Product Management/Intelligent Search and Retrieval, SER Solutions |
Bjørn Olstad, Ph.D., CTO, Fast Search & Transfer
Alkis Papadopoullos, Director, Linguistic Technologies, Convera
Dave Schubmehl, Vice President/Discovery Products, Open Text
Search is the most active segment in the information management marketplace, and the least understood. That was the starting premise for a wide-ranging series of communications that were recently exchanged between KMWorld and some of the leading vendors of new search technologies. Boiled down to this series of seven key questions (and answers), these conversations reveal that both new insight and old-fashioned common sense are alive and well in the search business.
1. Enterprise search more or less exploded as a market segment in 2003-2004 or so. To what do you attribute this sudden increase in market activity, from both the customer side and from the vendor perspective?
Jim Baum, Endeca: There are many reasons for the rise of enterprise search over the last year, but perhaps most significant are (1.) a growing need to realize the full potential of previous enterprise investments; (2.) the proven ROI of search in key application areas; and (3.) a step forward in quality, and expectations, as new innovations in search technology finally hit the streets.
Steve Kusmer, Atomz: User expectations of search have increased over the past few years. As a result, vendors have moved to beat those expectations. Now a great search solution requires guided search and navigation, categorization, deep linguistic support and fine-grained results control.
David Luzier, SER Solutions: With a better understanding of the power of search solutions, many are realizing that the search capabilities embedded in their existing applications (document management systems, portals, etc.) fall woefully short. Organizations are now better-educated and demanding better solutions from their vendors.
David Bercovich, Google Enterprise: It's easier to find the average rainfall in Nepal than it is to find basic information inside your own company! This is a strange paradox and also an opportunity. Search within the enterprise should provide the same fast, high-quality results that users find on the Web.
Jim Baum, Endeca: Recent (circa 2000) six- and seven-figure enterprise investments have given organizations a new wealth of information that SHOULD help with better decision making and improved business processes. But they fall short of expectations due to problems with access and usability—creating an abundance of shelfware and leaving users helpless to unlock their full potential.
Alkis Papadopoullos, Convera: The hype around the Google IPO has added a spotlight to the notion of search, but it is more likely that the role search has taken in the self-service market to support customers better, increase customer satisfaction and facilitate up-selling and cross-selling has helped to push search through to a different market segment.
Andy Feit, Verity: There has been a lot of new technology, but the customer remains pretty cautious in his/her IT spending. It continues to be a buyers' market, with many vendors pursuing the right to solve the customer's need—generally a good thing for the customer, and a challenge for software companies.
Dave Schubmehl, Open Text: There are three complementary forces that seem to be at work:
1. The amount of unstructured information is dramatically increasing, from increased records storage, mergers and acquisitions and the proliferation of corporate e-mail;
2. Corporate and fiscal compliance applications are increasingly in demand. Search, classification and notification are necessary components of a corporate compliance strategy; and
3. Senior executive awareness of search is at an all-time high due to the "Web search" effect. They see how useful these technologies are in providing answers from the Internet and they want the same or better for their own organizations.
Bjørn Olstad, Fast Search & Transfer: Search has moved from the action of last resort to the default way to find and discover information.
2. Do you think your customers view enterprise search as a "cost-saving" tool or a "value-added knowledge-support" tool? In other words, is search a "must have" or a "good-to-have" technology?
David Luzier, SER Solutions: If you were to ask this question a few years ago, the answer would have been that enterprise search was a "nice-to-have" technology. Today, organizations understand how much time and money can be saved within an organization when employees are able to quickly and easily locate relevant information.
Steve Kusmer, Atomz: If a business case can be made—improved revenues or improved employee productivity—then enterprise search goes beyond being justified by its total cost of ownership and instead can be justified by ROI. It is much more satisfying to help improve the customers' revenue lines rather than lower their costs.
Bjørn Olstad, Fast Search & Transfer: There are two distinct groups: First, there are customers who see search as a competitive advantage in leveraging information internally and externally. The second group is looking for a quick, low-cost way to just index documents—i.e. "just find them," not "mine them."
Jim Baum, Endeca: One of our Fortune 100 customers, when asked about the ROI of their new intranet search application, said "it may not be the business metric you're looking for, but we have noticed a 175% boost in the use of search." And he was right; it was better than any ROI number we could have expected. It meant we were delivering something that had been sorely missing.
David Bercovich, Google Enterprise: Web search has opened up everybody's eyes. Nobody has the patience to click through endless folders hoping to reverse-engineer somebody else's concept of where a document should be categorized. It's hard to beat the simple joy of typing in a few keywords and getting back exactly what you're looking for.
Alkis Papadopoullos, Convera: The heavier focus on how Web search has enabled employees to be more information-savvy, coupled with the emphasis on the importance of "mission-critical" search-based applications, are together pushing us much closer to "must-have."We're getting far fewer information-gathering RFIs that were primarily serving to educate customers. It is now clear that the customer is thinking: "which search technology to adopt?" rather than whether to adopt search.
Andy Feit, Verity: Some companies view search as merely a solution to end users' complaints. But the enterprises that are more aggressive in getting the most value out of their knowledge, either because it is their primary asset (for example, in a research organization, or a consulting company) or because it gives them a competitive edge (better customer support, faster time to market) demand much more. These customers cannot imagine life without it.
Dave Schubmehl, Open Text: For corporate and e-mail compliance, and other "bet-the-business" applications, search, classification, entity extraction and information clustering are "must have" technologies.
3. To what degree has regulatory-compliance pressure affected your business? Do your customers consider "search" a critical part of their compliance?
David Bercovich, Google Enterprise: Regulatory compliance has resulted in search projects that are mind-boggling in scope. The task of retaining e-mails and documents across tens of thousands of employees for years is difficult to imagine. The further task of retrieving a single important document from that archive sends chills down your spine...
Dave Schubmehl, Open Text: Customers are looking for a comprehensive solution that includes collection, monitoring, search, classification AND records retention. They also want technology that can deal with terabytes of unstructured information and deal with it quickly and effectively in a secure environment.
Andy Feit, Verity: Search and classification must be part of a greater business process; what you do with the information is more important than just finding it. This is the domain of business process automation and BPM. Task-based interfaces, tied into the business process, can help users along the way with the information they need for their roles in the process, all proactively.
Bjørn Olstad, Fast Search & Transfer: The time it takes to gather documents for legal discovery without rich full-text analysis can make you miss the deadline.
4. Which is the more effective approach in your opinion: A pre-arranged taxonomy/classification scheme, or a robust full-text search tool? And why?
Jim Baum, Endeca: I spend a lot of time talking to "information scientists" (a more glamorous name for librarians) who were thinking about this problem long before computers showed up on every desk. Guided Navigation is based on something librarians have called "faceted classification" since the 1930s. Essentially, old-style taxonomies put "books" on a single shelf. The faceted approach puts the same book on many shelves.
Alkis Papadopoullos, Convera: There is a common misconception that these two approaches compete, when in fact they must be viewed as complementary.
There are two familiar "use cases" that need to be supported. The first is when the user knows precisely what he is looking for and can express it well in a query. The second involves the need to explore information. "I need to answer the following specific question" versus "I need to obtain information about a topic." Both examples might be used by the same searcher, depending on his task. Directed search and browsing should therefore go hand in hand.
But in the marketplace, these two approaches are contrasted against each other with a quasi-religious fervor.
Andy Feit, Verity: There are times when you just want to see what information assets are out there, or perhaps you aren't sure how to describe what you're looking for in simple search terms. For example, in every company's HR organization there will be common information on, for example, "vacation policies." A search on that term would work just fine.
But, what about special benefits that your company has for health club membership, or discounts at local attractions, etc.? The user would never think to type those terms in. But, a directory—or taxonomy—of HR information, organized by topic, would have interesting categories that employees could browse, often discovering information that is of value.
Bjørn Olstad, Fast Search & Transfer: Clustering can reveal insight in the results that a user can instantly relate to but did not know how to enter the query to get to the results.
David Luzier, SER Solutions: Taxonomies are time-consuming to build and difficult to maintain and change. Ultimately, the goal is to provide users with immediate and relevant information. Since this can be achieved without the need to build and maintain taxonomies, robust, full-text search solutions that search for content, not just keywords, are a much better approach. Dave Schubmehl, Open Text: A good taxonomic scheme provides clues and context about the content that the user is working with and makes searching and analysis more effective by quickly letting users "drill down" to the information they're seeking...or, more importantly in compliance applications, find out if that information is not in the repository.
5. To what degree is "enterprise search" a front-of-mind item in the IT departments of the customers you deal with? How about among the line-of-business people you deal with?
Bjørn Olstad, Fast Search & Transfer: Line-of-business managers are moving quickly to specific solutions where they can define the ROI clearly for their business, whether it's gained from "selling more on the Web site" or from cost-savings.
Steve Kusmer, Atomz: The line-of-business people have the direct responsibility for the results of search, and are the ones who benefit. They need to be more in control of the search application.
David Luzier, SER Solutions: While finding information fast is important to line-of-business personnel, considering what alternatives are available to support this requirement is not front-of-mind for them. The IT organization is responsible for addressing the enterprise search needs of the firm and is therefore the primary driver.
Jim Baum, Endeca: The line-of-business people usually write the search RFPs. They're the visionaries who see how finding and reusing information can transform a business process.
IT sees enterprise search mostly as a plumbing problem. They only worry when they hit scalability or performance walls, or when hardware gets too expensive, or administration becomes too painful.
David Bercovich, Google Enterprise: Historically, enterprise search projects have been complex, resource-intensive and plagued by low adoption rates. We see a fairly radical shift underway as companies realize the incredible value of enterprise search and that—unlike many other IT priorities—the search problem can be solved quickly without breaking the bank or stretching the IT department.
Andy Feit, Verity: The more progressive IT departments are reaching out to their constituents, defining standards for search tools and corporate-wide taxonomies and ensuring solutions fit with the overall enterprise application architecture (e.g., factors such as security or Web services).
Dave Schubmehl, Open Text: IT departments are recognizing that there is a qualitative and quantitative difference between the search technologies needed to index and organize simple applications like corporate Web sites and the technologies needed to find, index, federate, organize and extract information from all of an organization's repositories, shared drives, corporate e-mail servers and databases.
6. What is the next challenge for the search-vendor community? Could be anything technologically you're looking at downstream (Messaging? Voice? Video?) or it could be market and economically related...
Bjørn Olstad, Fast Search & Transfer: The next challenge is the mobile user, and the need to apply "contextual insight." For example, if the query is "BBQ" from a mobile phone, the search needs to remove BBQ restaurants that are too far away, or are closed at this particular hour. It then needs to know the type of phone, so the results are presented in the best possible way.
Alkis Papadopoullos, Convera: There will be a very strong push for search to become a more integral part of the sharing and disseminating of information, and search applied to business intelligence to provide directed background data and search "reporting tools" to support and improve the decision-making process.
Also, ontologies, taxonomies and semantic networks will take on a greater role in helping to push relevant content to communities of interest, teams and business units ...in essence, push will start to take on much more importance than the current pull-based model.
Steve Kusmer, Atomz: The integration of Web analytics with site search and Web content management applications is fertile ground for innovation. It's a new era, where search and analytics combine to optimize the visitor experience.
Jim Baum, Endeca: These are exciting times—we're seeing the erosion of the long-standing walls between databases and other content repositories. This makes it possible for users to find content that was previously siloed in different enterprise systems.
Although that technological problem has been elegantly solved, there's a big lag in designing applications that unlock the full potential. Part of this lag is due to politics—the people who own the silos don't write RFPs for new systems to cross them.
But part of this lag is because people are still dreaming up new ways to use the technology. Literally every week, customers show me visionary new applications they have built on our platform—applications that are still unnamed that creatively solve business problems that were never addressed before.
7. Tell me what motivates your company the most about the enterprise search market and the opportunities it represents... in short, what's so hot about search?
Bjørn Olstad, Fast Search & Transfer: Search is evolving to deliver contextual results. A decade ago, people would say, "why doesn't the computer do what I mean versus what I say?" We are getting close to that now.
Andy Feit, Verity: This market continues to offer room for innovation. Other software segments are much more static. Databases, for example: How much has really changed in how we do this? Sure, it's become more scalable, or added a few new operators, but fundamentally it hasn't really changed much, and probably doesn't need to.
Alkis Papadopoullos, Convera: There are few things as satisfying as realizing that you have just shaved off 50% of effort to write a report because you located previous work done on the topic...or ensuring that a market analysis report includes detailed competitive information because you were able to access resources and content...or suddenly realizing that two seemingly unrelated concepts were in fact interacting in a very unexpected way that is critical to your understanding...
Dave Schubmehl, Open Text: Search, classification and related technologies are the glue that holds ECM systems together. The ability to effectively find and utilize that key piece of information is how ECM systems deliver their capabilities.
Jim Baum, Endeca: People STILL can't find what they're looking for! Every time I'm at a social gathering, when people ask what I do and I mention search, the floodgates just open and they pour out stories about information overload. What's so exciting about my job is that we have a genuinely new, elegant way for people to find things.
Search is in just the first few innings of the ballgame. I can already see part of how the next few innings will play out by looking in our own labs, but there's still so much opportunity, so much room for improvement, so much of the game left to play.
Andy Moore is a 25-year publishing professional, editor and writer who concentrates on business process improvement through document and content management. As a publication editor, Moore most recently was editor-in-chief and co-publisher of KMWorld Magazine. He is now publisher of KMWorld Magazine and its related online publications. As Editorial Director for the Specialty Publishing Group, Moore acts as chair for the "KMWorld Best Practices White Papers" and the "EContent Leadership" series, overseeing editorial content, conducting market research and writing the opening essays for each of the white papers in the series.
Moore has been fortunate enough to cover emerging areas of applied technology for much of his career, ranging from telecom and networking through to information management. In this role, he has been pleased to witness first-hand the decade's most significant business and organizational revolution: the drive to leverage organizational knowledge assets (documents, records, information and object repositories) to improve performance and improve lives.
Moore is based in Camden, Maine, and can be reached at andy_moore@verizon.net.