Putting intelligent search to work

Cognitive search The implications of this trivial example are considerable for enterprise search users. It indicates that effective search is based on understanding basic attributes such as who the user is, what he or she is looking for, and how that meaning is conveyed in the specific language of searches. All of this knowledge initially stems from organizations’ end users, their terminology, and specific business objectives. “You always start with a question from someone in the business, because these NLP [natural language processing] people have no clue what is important to the business,” Franz CEO Jans Aasman said. Frequently, the various knowledge about users, their goals, and the hierarchical concepts relevant to them are stored in knowledge graphs.

Secondly, cognitive search involves analytics—not just indices and basic retrieval—which is integral to natural language generation systems. “The right analytics in your system lets you answer more and more questions,” observed Neil Burnett, CTO at Arria NLG. “In some cases, it’s all just purely statistical stuff. Then, there are more advanced approaches—not just for simple outliers but [for identifying] what’s an anomaly in context or over an extended period of time.”

Finally, there’s an immediacy to cognitive search solutions for rapid question-answering, some of which foregoes lengthy (and costly) machine learning model training or manual taxonomy building. Users can simply feed such a system a corpus of documents they want to search, Welsh said. “As soon as it ingests the documents, which—depending on the data size— can be a few minutes to a few hours, you can immediately start asking natural language questions.”

Conversational AI

The conversational nature of cognitive search has obvious conversational AI implications. This AI variety involves the gamut of NLP technologies, including natural language generation, querying, and understanding. Synthesizing these technologies results in a “system-led conversation with the individual so that they may start off with a question, and, through a process of disambiguating their question, get an answer,” Potter noted. “That’s really an element of a conversation.” Contextualizing an individual’s questions with his or her attributes and interests is critical to these conversational interactions. According to Burnett, the capacity for intelligent search platforms to comprehend the overarching context of questions enables users to “find something interesting and drill down into more details.”

The notion of collective intelligence is pivotal for determining user context and allowing systems to guide conversations along inquirers’ interests. “Where you’re really going to start seeing the value is in this idea of leveraging what everybody else is doing around you because that collective intelligence is adding increasingly valuable context that actually increases the hit rate,” Potter said. In addition to guiding individual searches or questions into conversations, conversational AI makes the need for detailed responses via formal queries obsolete. “The idea is to get away from programming languages and get computers’ input for search using techniques that are easier for people,” Pingali noted.

Intelligent search

The cognitive computing underpinnings for conversational interactions with search systems are as broad as this discipline itself. Rapidly parsing natural language questions for apposite search answers involves a variety of approaches. “It’s speech recognition, taxonomies, entity extraction, rule-based processing, reductions, and machine learning,” Aasman said. “It’s a mix of statistic and non-statistic approaches—because you care about what you’re solving; you don’t really care what solves it.” These are some of the plentiful approaches involved in intelligent search:

♦ Deep neural networks: These machine learning models epitomize the statistical method Aasman referenced. “Word embeddings started in 2012. Now, we’re coming up to the transformer models like BERT which are the current state of the art in machine learning,” Welsh said.

♦ Symbolic reasoning: This form of AI exemplifies its non-statistical side and is predicated on defining taxonomies, employing ontologies, and arraying information on knowledge graphs for intelligent reasoning (as opposed to learning) with rules.

♦ Neuro-symbolic AI: This approach blends AI’s statistical and non-statistical foundations to deliver concrete knowledge for reasoning with the pattern detection prowess of “a purely learned perspective to give a richer understanding and semantic understanding of language,” Welsh explained.

♦ Computational linguistics: This discipline enables cognitive search systems to “choose the right words to articulate findings properly,” Burnett said. “It describes the patterns that have been detected, but in a way that contextualizes them so the reader will understand and get the meaning without looking through lots and lots of graphs, which may be confusing.”

♦ Bots: Software agents can serve as an interface for user interactions or provide back-end analysis of data for searches in what Aasman referred to as “reductions.” “We use bot techniques to analyze what’s being said,” Aasman commented. “We analyze the questions asked and the answers given, and we try to simplify it so it’s easier to do analytics on it.”