Today’s search technology, sometimes now referred to as insight engines or cognitive search, is much more intelligent, capable of returning concept-based results rather than simple word matching based on indexed content. However, users still do not think enterprise search “works like Google,” which is often the stated goal. This desire is based on wanting a simple search interface, highly relevant results, and the presentation of a context-sensitive set of information. “Users want to see an answer, a knowledge card, a snippet that presents the most relevant excerpt, and (sometimes) locational data,” observed Khan. “We make it look like a Google response, but with a much lower resource burden.”
Pureinsights uses a combination of open source search technology such as Elasticsearch, Solr, or OpenSearch; NLP, such as BERT; and graph data- bases such as Neo4j to integrate and present the results most likely to be responsive to the user’s query. Since graph databases represent relations or connections among entities, they are particularly strong when it comes to presenting answers to questions. “When all these work together,” Khan added, “you have AI-based search.”
Vector search represents documents and query terms as calculated mathematical expressions in (multidimensional) space. The closer the document and query are in the vector space, the more likely the document is to be relevant to the search. Semantic vector search is particularly helpful in dealing with ambiguity, such as when a word can have different meanings depending on the context, or situations where there is the need to find the right answer when different search terms with similar meanings are used. Resolving these issues can improve the ability of search applications to detect intent.
Models are available that are already trained on large amounts of data, and techniques for training deep learning vector models specific to a set of products or queries can be used to optimize search results. “Vector search is another way to surface content,” Khan pointed out. “However, classic index search is still important.” With improved techniques for finding the content that users need, enterprise search may become a more compelling asset.
Although Pureinsights offers the typical professional support to companies that want help starting up and then choose to manage their own search function, its primary business model is based on the premise of full-service search. “If search is critical to a company’s business but is not a core competency,” said Khan, “a strategy of working closely with a team of experts that is responsible for this function may be the best approach, given the challenges of hiring and maintaining an in-house team.”