AI’s Impact on Data Silos and Knowledge Hubs
According to Aasman, since a language model is part of the knowledge graph stack, “If you ask it a question, it will say, ‘What does this mean in terms of the enterprise technology?’ Then it says, ‘OK, how do I implement that independent of the database technology?’ It will say, ‘I need this, and this, and this, from this database.’ Then, it will write SQL queries, Parquet queries, DuckDB queries, etc.” Thus, users are not only able to issue queries in natural language, but they can also prompt the system to perform 10–20 queries to arrive at a satisfactory response—all by asking one question.
Vector Computing Engines
The vector database approach Bixby mentioned provides sophisticated information retrieval results, particularly when intelligent chunking strategies are involved. Chunks are the discrete pieces of enterprise content that comprise vector embedding. Competitive solutions have algorithms that chunk enterprise knowledge for users, so that “the accuracy and the summarizations of the questions across that same knowledge are so much vastly better, in terms of pulling back relevant chunks and then summarizing it, so that you don’t have to read every single article the search found,” Bixby noted.
The algorithms select from multiple chunking strategies, including overlapping ones (in which parts of the vectorized content appear in more than one chunk), and those predicated on a document’s sections or bullet points. This approach is pertinent for “looking at similarity or trends between purchase orders or vendor items,” Pava remarked. “Or, you can use it to do basic reporting: if you put forms and processes in and say, ‘What are the trends of these? What are the common items?’”
Information Retrieval
No matter which form of information retrieval an organization adopts—including vector similarity search, federated search, and lexical search—search has always been one of the foremost constructs for knowledge access and sharing. The capacity to search across the content in data silos is invaluable. With some approaches, it’s necessary to collocate the information contained in silos or knowledge hubs.
Several KM vendors have tooling so that when users “want to create a new corpus of knowledge, you point to that thing, whether it’s a SharePoint site, or a series of documents, PDFs, word documents,” Bixby said. “It could be exterior websites, anything. And it will bring that knowledge in so you can search it.” With this method, even though there are silos, organizations can consolidate their information into a single repository for universal access.
In other cases, information retrieval is a precursor to, or enabler of, the mitigation of silos across knowledge hubs. Traditional keyword search is still a form of AI, despite being overshadowed of late by other types of information retrieval. Pava mentioned that keyword search may be useful for obtaining the knowledge necessary to facilitate meaningful natural language interactions that require advanced ML. With this paradigm, classic search “is about finding where content lives,” Pava said. Such content is oftentimes dispersed across repositories that might be siloed.
However, once germane content is located via search, it’s no longer necessary for a human to “open it up, interpret it, and see if it’s relevant to me or not,” Pava added. “Conversational chat capabilities help by letting you ask questions of the content. Not just, ‘Where do I find it?’ but to get insights from it directly.” Vendors often couple the respective constructs Bixby and Pava mentioned. Consequently, organizations can load content into KM repositories, which automate the embedding and chunking process. When that is in place, users can then ask questions in natural language, get summaries, and reap all the other benefits language models provide for information retrieval.