Social Networking and Open Source Search

Unless you've been living off the grid in a cave, you've experienced how social networking and social media have penetrated our society. But if your organization is like most others, you're still wondering how best to bring social media inside the firewall—whether it is crowdsourcing, microblogs, self-organizing virtual teams or other hyper-connected, always-on (but not always productive) tools. Many companies are trying social technologies outside the firewall, too—trying to better leverage partners, suppliers and customers as they tweet, check in and express whatever sentiment occurs to them at the moment.

There is, of course, no one right strategy—approaches will be as different as businesses are from each other. One thing is clear: Search and related tools are an essential piece of any social strategy. This new social search goes well beyond keyword lookup in "traditional" enterprise search, beyond crawling the corporate content management system.

More than ever, search technology needs to be fast, efficient and cost effective—especially considering that even in a medium-size company the number of social interactions (multiplied by the amount of content) one might be interested in can quickly bust through arbitrary document limits imposed by conventional search software licenses. No technology solves these problems of functionality, cost-effectiveness and scale as well as open source search.

Social media giants LinkedIn, Twitter, Yelp and many more all run search on the open source stack of Apache Solr/Lucene. It delivers the scale and speed essential to such applications. Perhaps the most important attribute is its inherent flexibility, so you can incorporate best-of-breed technologies or readily fill in the gaps unique to your organization. For business-critical implementations, commercial-grade support and packaging are essential; now, you can use Solr/Lucene in an enterprise-class package (such as LucidWorks Enterprise) for commercial-grade search application

Connecting the Dots

One particularly effective way to tie search and social networking/collaboration into the enterprise is by combining content (what people are talking about) with the social graph connecting people working on that content (who they are talking with). The first step is to combine indexing of traditional content typical of "enterprise search" with internal social collaboration content such as instant messaging, blogs and microblogs, email and the like. With open source Solr/Lucene, you can use the same technology to harness information about how users interact with all of that content and with each other—to deliver a social experience with more focused interaction, serendipitous discovery and better productivity.

For instance, it is often the case in any organization that many people are trying to solve similar problems, unbeknownst to each other. With the right social search and discovery tools, they could likely solve them better, faster and cheaper. (As an aside, I once met two people from the exact same large company at a conference, five minutes after one another, who were working on the same problem and had never met.) In large companies, simply asking or involving one or more existing experts on a topic within the company often solves the problem at hand, if only you could easily find the experts.

The old way to solve for this would rely on a search engine that indexes the company directory where employees self-identify what group they are in (or rely on word-of-mouth: "You should talk to Bob in widget design, he's solved that problem before") to find the right person. The modern approach uses the search platform to index and analyze both content and users' social graph–along with support from big data analytics tools (complementary tools from the open source stack, such as  Apache Hadoop, Hive, etc.). In such a system, the search platform provides the core capabilities of indexing the content and connecting users to those results in dynamic ways through their queries and questions. Offline analytics churn through the content and user interactions with the content, feeding it back into the search system. This way, users can search, browse or otherwise be alerted of both content and users that can help.

Another thing that I routinely see in such a system is the need for a relevance model beyond simple keyword matching. Such a model combines scoring factors for the content (i.e. traditional search matching) with user profile information (title, location, business group, resume, etc.) and social connections. Such a relevance model scores keyword matches as usual, but then adjusts scoring based on what organization the user is in (an engineer is more likely to want to see engineering documents than marketing documents that have keyword matches) and who the user is connected to in the company social graph (perhaps favoring content that was also read by her peers or her boss).

Adding users to the feedback loop between relevance and findability is another way to bring social sophistication to search, both customer facing or internal facing. Many Web applications typically ask for ratings and reviews that loop the feedback into the search system. We added automation to this feedback using our "click scoring" mechanism. It automatically captures what results users selected, and can recalculate relevancy based both on the value of the content computed by the search engine and empirical input on what users find most valuable. This approach can easily be extended to many elements of the search experience ("did you mean...," facets/navigation, upsell/cross-sell/related items, paging controls and even query rephrasing). In this way, users need not go out of their way to provide feedback (ratings/reviews) but instead vote with the actions they perform routinely as a part of search.

While it's still early for social networking in the enterprise, there is little doubt that search is essential to making the connections between users and content—and to each other—more transparent across the social landscape. Many of the best ideas are yet to come, with tremendous opportunity to leverage search in new ways. The challenge to your search platform will be to sustain the flexibility and transparency needed to embrace these new opportunities.