Four Tips for Evaluating Search Tools
The word “intelligent” is used so often in describing KM applications, it can be challenging to evaluate search tools to determine whether they are best for a particular set of needs. Here are four useful ways to evaluate a toolset’s ability to understand, reason and learn, and the types of interactivity it can enable:
- Understanding—The ability to process user context and input in rich, actionable ways.
The more relevant inputs a search can process, the more targeted and precise its results are likely to be. The search query itself can be processed using word stemming and synonym association, as well as deeper natural language understanding of important language patterns. In addition, other elements of context can be applied to quickly and accurately narrow the domain of interest that the query applies to. Categorization metadata, user identity and recent behavioral data can all bring the right responses into sharp focus.
Look for: Tools that can process both context and query input in rich ways, and allow for the efficient evolution of these definitions such that knowledge always stays in synch with evolving questions, terminology and user scenarios.
- Reasoning—The search tool’s techniques and capabilities to collate, assess and prioritize the best response to user input.
There are multiple reasoning mechanisms layered into any sophisticated search algorithm, including word/term frequency and proximity, weightings of specific information sources, and linguistic identification of word patterns, etc. Each search technology uses its own approaches and prioritization of what drives best results. Often, basic search testing can be the best way to confirm the results of reasoning against a particular set of content, given what users are likely to do and expect.
Look for: A clear expression of reasoning mechanisms, and how they can be evolved over time. “Black box” search approaches may be powerful but need to be validated in testing and maintainable against large, evolving collections of information in real-world scenarios.
- Learning—The mechanisms by which the search system can self-optimize based on user behavior.
Machine learning has increasingly broad applications as it is applied to more scenarios in which calculations of behavior can apply to future responses. Standard “ML” in search refers to the ability to associate knowledge objects with specific query contexts, and promote these likely results for future similar requests. ML also has more sophisticated application in mapping user behaviors as inputs, such that user behavior patterns before or during their knowledge interaction can be used to calculate the best information needed.
Look for: How machine learning is applied, and how deeply it can be leveraged. Can machine learning be applied to contextual inputs to drive deeper responses to next-best actions? Is there visibility into—and some control over—where ML is applied within a toolset, to tailor and evolve the fit to user needs?