Don't Curtail the Long Tail: Lessons for Search and Information Access
The "long tail" theory says there's latent demand for each piece of information in your content and inventory collections popular ones. And with the reach of intranets, portals and the Internet, it's now possible to satisfy the long tail of demand out to the very end. But it takes more than simple search to help a person find the content that matters to him and no one else. So how can anybody find anything in there?
Out of One, Many
The long tail was first described by Chris Anderson, editor-in-chief of Wired.1 Anderson points out that the assumptions we use to make scarcity-based decisions in the offline world are turned upside down by the Internet. Online, it costs practically nothing to add another piece of content to a collection or another SKU to a supply chain. It consumes no additional shelf-space because online shelf-space is infinite. It doesn't have to be held in on-hand inventory because it can be sourced at the time of an order. Similarly, serving one more user costs practically nothing. Even if taking on 10,000 additional users incurs additional costs in hardware and bandwidth, this pales in comparison to the outlay for construction, staffing and supply infrastructure for new buildings. And the potential benefits are enormous.2
Don't Cut the Long Tail Short
The long tail is simply a power law rule. Power laws crop up all over. Take words in the English language, for example: A few get used all the time (e.g., the, of, and), some occasionally (e.g., today, mother, probably), and an enormous number almost never (e.g., conquistador). This is a power law. Page visits on a website also form a power law.3 In fact, every search or navigation log we've looked at shows a power law apps. (See figure on page S4, KMWorld Government Supplement, June 2006.) The common elements in all these cases are freedom of choice combined with a large number of options. Inevitably, inequality arises as more people, influenced by the choices already made by others, converge on a few selections. But the convergence is never complete. Each individual has goals that most others don't. A few constituents want to read Grover Cleveland's inaugural address. A handful of workers need the dental plan exemption form. And only one intelligence analyst needs to know who flew into Nome, Alaska, from Yemen in July, 2003.
Historically, agencies couldn't support all these one-off goals. They had to cut the long tail short. Now, they don't.
More Needles, Bigger Haystack
But, of course, there's a price. Capturing the long tail opportunity means offering an abundance of information, which becomes overwhelming when a person can't find what he's looking for. Searching looking for something, whether with keyword search or other means a scenario of small but difficult choices. The sheer number of scenarios in the long tail makes optimizing each one impossible. To provide the best experience for each user, focus on the three principles that drive all search scenarios. When people search, they:
Predict: With each step, the user tries to predict the likely effects of his action. But the inherent uncertainty of seeking something not yet found makes these predictions difficult. For example, to find the dental-plan exemption form, should a worker click "Dental Plan," which could have all the plan information, or "Forms," which could have all forms, regardless of type? Or should he put in a keyword search? And, if so, what if he calls it an "exemption form" and they call it a "benefits declination" form? If he makes the wrong prediction, he's wasting time.
Adapt: As people learn something more about the thing they're looking for, the functionality of the system they' re using, or the content in it, they modify their approach.4 If a user were at an online job posting site looking for a procurement position near her home town, she might start with a search for "procurement." When she looks at the thousands of results, she sees an option to refine the list to just her town. She clicks it. These are two completely different methods of searching. But people see them as perfectly complementary.
Iterate: The notion that people won't click more than three times is one of the great myths of website design. Research from User Interface Engineering dispels this myth by showing that users will take as many as 25 steps to reach their goals.5 The quantity of clicks doesn't matter as much as the quality. As long as people make progress with each step, they feel that their efforts are well spent.6 But if they don't see consistent progress, they're likely to give up.
Revise: Serendipity will take its course. People are likely to learn something completely new along the way that causes them to revise their goal. When this happens, they form a new goal and repeat the searching process.
Better Tools Unlock Searching Skill
Human searching behavior is not a problem; it's an untapped resource. But search tools to date often restrict these talents instead of enhancing them. To manage choice in the long tail, people need a solution where: