People Judge Relevance. Machines Calculate Evidence

Does the man make the machine? Or the machine make the man?

A pilot can't fly without a plane. A doctor can't diagnose without a centrifuge. A photographer can't create without a camera. And except in the simplest circumstances, the reverse is also true. But why? Couldn't you break down flying, diagnosis and photography into a set of rules that would allow the machine to make the same decisions the person would have made if he had been there? No. There is something more to the human decision-making process than simple logical rules can account for.

This fact is a central problem for traditional information-access technologies, including keyword search, databases and business intelligence (BI). Each of these "machines" intends to help an individual find the information she needs to make a decision at a unique point in time. This is one of the most difficult technological problems of our age, and one of the most valuable to solve, but too often we're asking the centrifuge to do the work of the doctor. Instead we should consider each person the expert and provide more effective tools for reaching their goals. But how do we evaluate which tools are the right ones for the job?

Is Relevance Irrelevant?

The familiar framing of the information-access problem is how to help people find the right information at the right time. But we need some measure of "rightness" before we can characterize the tools we need. "Relevance" is the technical term for the central measure of "rightness" in information retrieval. Unfortunately, relevance defies logical rules. You know this if a search engine ever returned to you a document that was "72% relevant." This is like a flight computer saying the plane is 72% on course.

Relevance is subjective because it's relative to the person's goal. For example, if I tell you that the movie tonight is at 7:30 PM, is that relevant? If we're going to the movies, then yes. If not, no. How can a piece of information be both relevant and irrelevant? It depends on what the user is trying to accomplish. Relevance is in the eye of the beholder.

Just because relevance is subjective doesn't mean we must give up hope in finding it. On the contrary, it must be the central focus of information access technology. But it does reveal why we shouldn't ask the machine to determine relevance for us. It can't. Fortunately, people are experts at judging relevance—and with help from machines, we can do better than without. The central function of information access technology is to produce evidence that informs human judgment. We summarize this as:

• People judge relevance
• Machines calculate evidence

Context Informs Relevance

So what can a technology do that would help people better determine relevance? First, let's look at how context helps people determine meaning. Take this simple exchange:

Host: "Would you like some dessert?"
Guest: "It would fill me up."

How does the host figure out if the guest wants dessert or not? If the host knows that his guest doesn't like to sleep on a full stomach, then the answer is no. If the host knows that his guest likes to eat until satisfied, the answer is yes. This context lives outside the literal dialogue in the mind of the host. It completes the proper meaning of the guest's statements. But how does the guest know the host will make the proper interpretation? He's actually relying on the host to have this context in mind and, therefore, interpret his statements accurately.

This is how human communication works. It's based on the resolution of ambiguities through best efforts and educated guesses. But both parties have to be working with the richest context-processing machine available—the human mind. Both guest and host use observations about the current environment, knowledge of the audience and personal experience to attempt to communicate effectively. The vast majority of the time it works. And the application of these contextual facts, taken from the environment and the head, is the key.