Discovering Hidden Information Treasures in ECM

Oil might sound expensive today, but in the mid-1800s, it cost around $2 a gallon, equivalent to about $200 in today's money. But that was a different type of oil: whale oil, hundreds of gallons of which are stored in the head and blubber of the whale. The market for petroleum had not yet been developed, and whale oil was prized by the wealthy for burning nearly odorless and smoke free, compared to the stinky and smoky—but cheap—tallow candles used by everyone else.

Whale oil was expensive because whales were extraordinarily difficult to locate. Expeditions spent multi-year voyages crisscrossing known migratory paths, waiting for a stroke of good luck when one of these mammals had to surface to breathe. (It took Ahab a dense 500 pages to find Moby Dick, which felt not unlike an actual hunt.) Nevertheless, as prices increased, the animal was systematically hunted to near extinction.

But as we know, petroleum became plentiful enough to displace whale oil. After the first oil well was dug by Edwin Drake at Oil Creek in western Pennsylvania in 1859, prices plummeted. Before long, petroleum prices sank to seven cents a gallon, and the whale oil market collapsed. That's not surprising, since it's clearly cheaper to gather oil from a sedentary supply.

Then, just like today, drilling for oil also relied on strokes of luck. The first wells were drilled near naturally occurring brine wells near Pittsburgh which already supplied a thriving salt industry. Oil seeped naturally into a nearby creek, and Drake drilled there in the manner of the briners. Explorers began looking for their next strikes near other naturally occurring oil seepages in streams. But before long, those had all been tapped and exhausted, and new supplies had to be found. And just like guessing where the whales were beneath the ocean, underground oil became hidden treasure, and the price increased. Wildcatters began drilling blind, hoping to strike oil.

By now, you should note some strong similarities between these early oil hunters and people today finding information treasures in your ECM systems. But this is more than just a surface similarity. As it turns out, contemporary oil explorers have techniques for finding hidden oil that also literally work for finding hidden information. The key problem we share with whalers and wildcatters is this: How can people discover things for which the precise location is unpredictable? And the new problem: how do we, as an information provider, serve an audience looking for something fundamentally unpredictable?

Drilling for Oil... A Better Way

"We spend tens of millions of dollars on data, and use it to make decisions worth hundreds of millions of dollars," says Sam Clayton, knowledge management specialist at BHP Billiton Petroleum, a division of the world's largest diversified resources company.

Several of Endeca's information-access platform customers, like BHP Billiton, explore for natural resources, drilling for oil and mining for gold by finding information treasures. Now that they can't follow oil seeps or known whale migration paths, they've found new ways to generate evidence of where to look. Teams of geoscientists study data like seismological readings, ground-penetrating radar and fault-line maps to help predict where to explore. Records of tiny fossils reveal the likelihood of the existence of mineral veins or oil fields, yet all the while generating a potentially overwhelming volume of reports. In the case of

BHP Billiton Petroleum, these reports were capturing tens of millions of dollars worth of information, which analysts needed to repurpose as each new project unfolded. BHP Billiton was drowning in information, hindering a team of renowned geoscientists from focusing on their specialty of piecing together the lay of the land from scant clues. Enterprise content management had a clear role in supporting them by managing all this raw information for analysts, helping them upload it, monitor its integrity, audit its provenance and make it possible for everyone to find that information in any context. BHP launched a new initiative called the getKnowledge Project, featuring an information access platform from Endeca, complemented by a customized EMC Documentum system built by Blue Fish Development Group.