Reimagining SharePoint for 2011
SharePoint as an Information Discovery
and Retrieval Powerhouse

It is no surprise that many enterprises and government agencies have adopted SharePoint as their principle information-sharing platform. By incorporating the knowledge captured in MS Office applications, enterprise applications, databases, the corporate website and email, Microsoft has delivered a well-designed general-purpose information repository and collaboration center for the enterprise to leverage.

We continually hear from executives that while SharePoint is great for consolidating information from individual desktops into a single storehouse, they live with frequent complaints from users that finding what they need in this massive data store is often a task of Herculean proportions. It is not that it is difficult to type in keywords in a search box. Rather, it is the thousands or more documents that are returned that match their criteria. And, that the tools available for refining the result set based on document type or creation date, or other criteria are not always helpful.

What if SharePoint understood who each user is, and what they care about? What if “context” were introduced into every search? What if matches were made even if the exact words were not in the documents, but were inferred? What if each query could build faceted selections on the fly, based on the results set— without metadata? (A faceted search is one where the user can fine-tune the results by selecting a check box such as “only spreadsheets.”) What if SharePoint could turn a simple query into something that a subject-matter expert would construct, even if they are no expert themselves? What if SharePoint could help users uncover what “they don’t know that they don’t know?”

This is not a new concept. It is called semantic search. Semantic search, quite simply, is the ability for search to extract meaning and context from queries—just like humans do. It is a quantum leap beyond simple query-string matching.

While we’re playing the imagining game, what if we could take that semantic search ability and extend it beyond the enterprise? What would it be like to include not just information stored in SharePoint, but also information from SaaS applications, the Web, blogs, social media and beyond? What if users could get a 360-degree view on a topic like a particular customer, industry or product? Again, incorporating not just what we’re thinking (behind the firewall), but what everyone else is thinking?

Now, here is where it gets interesting. What if you could leverage SharePoint, (as the content-rich information store that it is), couple it with other information stores outside the firewall, and build gorgeous graphical information applications that are purpose-built for large groups of users. Imagine a “customer” application that could be the ultimate resource of any customer-facing employee. One where they could understand every interaction, every touch point and even what a customer may be saying to others about your products on social media? Now, imagine that app getting deployed to thousands of users (with the appropriate security policies in place) in just a few weeks.

In the case of the US Army, we worked with the Combined Arms Center (also known as CAC) in Fort Leavenworth, KS. The CAC is a major subordinate headquarters of the US Army Training and Doctrine Command, and is often referred to as the “intellectual center of the Army.” A critical part of CAC’s mission is to provide the US Army easy access to tens of thousands of current documents, as well as even more archival documents dating back decades. CAC uses SharePoint to enable collaboration to improve operational processes—but wanted to improve the search engine functionality to increase productivity. Even more importantly, the CAC was running multiple instances of SharePoint and wanted to facilitate semantic search across all of these instances—without a big hardware investment.

How Did We End Up Here?
Several years ago, we coined the term “search-based applications,” or SBAs, to signal a new approach to information discovery and retrieval. Combining the best of both enterprise search and business intelligence, SBAs are purpose-built applications that sit on top of information stores (like SharePoint). The reasons for this new approach can be best understood by looking at the Web.

Visitors to so-called Web 2.0 sites can learn more about a particular topic, more quickly, than almost any other means. By aggregating data sources and presenting them in compelling layouts that inspire making connections where they were not immediately apparent, these sites are a model for the information consumer. Want to buy a house? You can talk to a broker, but you can also do a lot of self-education at Zillow.com. Want to rent an apartment? You can check for listings on Craigslist, or you could see those same listings come alive on a Google map at Housingmaps.com. The question we set out to answer was—why can’t this kind of technology, one that helps people make connections and intuitive leaps—be available to the enterprise?

First, we looked at business intelligence, or BI. While BI is exactly the right tool for analyzing structured information, like that found in enterprise applications or databases, it is a very poor tool for working with unstructured information found in documents, spreadsheets, websites, etc. The other challenge with BI is that it is inflexible, since the underlying data is often pulled together from multiple sources through complex processes involved in data warehouses. These factors have contributed to limiting the adoption of BI to “power users,” rather than large groups of people.

Then, we looked at enterprise search. Here, we have the opposite problem from BI. Enterprise search is good for unstructured data, but does not do a good job with structured information. And, enterprise search has the problem alluded to earlier—too many irrelevant results that get in the way of finding exactly what the user is looking for. While some enterprise search tools allow information retrieval beyond the firewall, few have connectors to the kinds of social media sites that are now important for completing a 360-degree view on topics.

Enter SBAs. What was needed was a hybrid between BI and enterprise search—”search-based applications.” Within a single instance, an SBA can provide information ranging from quantitative performance metric dashboards, to operations reporting, to textual narratives and anecdotes—that might include images and links to supporting documents and videos, and much more. And, of course, SBAs leverage the semantic search capabilities discussed earlier.