The Faces of Intelligence
Long after most professionals leave the events and activities of the day behind, a new breed of knowledge workers begins the workday. These silent, unknown professionals interpret and collapse the entirety of the day's events into a clear, manageable flow of information that reaches the highest levels of decision-making around the world.
Their task is to produce intelligence—a unique combination of hardware, software, communications, information and human "wetware"—and process facts and factoids, rumors and judgments, opinions and evidence through the complex calculus of human reasoning. They make the chaotic intelligible, the inchoate coherent and the disorganized clear-cut. In the course of a year, they will gather and evaluate millions of different pieces of information. They will organize each information element into one or more of more than 3,000 distinct categories and select the most salient items for distribution to the consumers who depend upon their work. The specific information may vary from day to day, but the results always have the impeccable, clock-like precision and unflappable reliability.
So how do they achieve such productivity and accuracy? By a simple and clear division of labor. They organize into three distinct teams. The first is the "intelligence gathering team." These intelligence gatherers have the responsibility for crafting and maintaining the intelligence filters that feed information directly to consumers. They serve as the principal gathering tool for the overnight analyst team. In the parlance of the intelligence community, this team produces work for both intelligence consumers and other analysts.
The second team—let's call them the "intelligent assessment team"—is responsible for evaluating information items in their domains of expertise and judging and grading information for its salience or decisionmaking value. They produce interest-areas and topical collections of documents for daily distribution to their consumers. Every evening, these analysts evaluate, categorize and organize a stream of thousands of documents into a collection that's ready to be consumed by their intelligence consumers.
The third team is the one responsible for operating the real-time assessment and early-warning desk. In addition to the use of screens and filters prepared by the gathering team, this group—call them the "intelligence early-warning team"—monitors real-time information and traditional media services to alert consumers to important events or critical information. These analysts provide consumers with a knowledgeable desktop where a just-intime view of strategic, tactical and political forces is made available.
A Background for Intelligence
The backgrounds of our three intelligence teams are quite similar in both professional training and previous work experience. These common elements provide a shared framework for the analysis, assessment and early warning tasks they perform—creating a common perspective and objective within and across the teams.
On the gathering team, most analysts are professional information scientists. With experience in research, professional library and academic, these professionals really excel at devising strategies for uncovering and characterizing the neverending flow of content. On the assessment team, each analyst is a bona fide subjectmatter expert, with direct understanding of the issues, trends and noteworthy events. The early warning team analysts are professionals with significant experience in real-time, deadline-driven operations, taking pride in being the place where consumers get the "big news" before it is on CNN or MSNBC.
In this classic intelligence setting, these professionals are consummate "situation desk" professionals, ready and able to net out the most important information at a moment's notice. To understand what they do and how they can become more productive, we need an outline of the information, technology, and analytic context they work within. If we can get our arms around that, we would have a good understanding of what the next-generation intelligence systems should deliver.
Capturing the information. In order to provide up-to-the-second intelligence to their consumers, any organization requires an operational system that processes information in real time. It is almost akin to a content refinery, designed for content acquisition/normalization/analysis/storage at a rate of thousands of content pieces per day. In addition, the refinery is designed to process and deliver items for further analysis or immediate consumption with sub-second latency. Given the missioncritical nature of the information provided by the system, its design also requires explicit redundancy and fault tolerance.
Specifically, the system collects information from thousands of sources in a dizzying variety of data formats. The system normalizes each item, converting the item into a single, standard format and representation, and then passes the information off for a first round of sophisticated analysis.
Encoding the intelligence. Next, the system analyzes the "content DNA" of each item—assigning appropriate metadata to characterize the item and storing the item and metadata into a database. Once sequenced, each item is available to support the downstream work of assessment and early warning. The intelligence for sequencing the content DNA depends on the patterns constructed and maintained by the gathering team. The importance of maintaining these sequencing and analysis filters cannot be overstated. As concepts, entities, events and the phrases change, the content DNA of an item mutates. The gathering team performs its research, crafts and evaluates the strategies and tests filters to minimize the mismatch over time or across concepts.
Performing intelligence analysis. These normalized and sequenced items are available for the consumption of intelligence— immediate provision to consumers or early warning analysts, indexing for subsequent retrieval or processing for use by the intelligence assessment team.
The first use—namely, provisioning— requires both a real-time push notification and asynchronous pull and presentation methods. The second use utilizes sophisticated query and search methods for information retrieval and analysis gathering, with a scalable, high-performance search engine platform. Such an engine gives analysts or consumers the ability to perform more sophisticated analysis by crafting complex boolean expressions, restricting results by using information from proprietary metadata, employing proximity and stemming operators to narrow or expand meaning and domain restriction by source and date of issuance, to make association concrete.