zenpundit.com

CRAFTING BETTER ANALYTICAL INTELLIGENCE PRODUCTS

Excellent declassified article from the mid-1990’s Studies in Intelligence.

“ Making the Analytic Review Process Work” by Martin Peterson.

“The problem with the review process is not the layers of review but rather the quality of the review. In an imperfect business, this is the one thing that intelligence officers need to get right. My 30-plus years of experience leads me to conclude that there should be three levels of review and three broad areas of review for each piece of finished intelligence.

Editing is NOT review. Editing is a mechanical task that should be accomplished by the first-level reviewer or by a staff. Review is about thinking, about questioning evidence and judgments. It focuses on the soundness of the analytic points that are being made and the quality of the supporting evidence. Levels of review is NOT synonymous with layers of review. Layers of review speaks to how many cooks are involved with the broth; levels of review is about ascertaining the quality of the soup.

Each level of review has a different focus. The strength of the review process is directly related to the different perspective that each level brings, with succeeding levels focusing on ever broader issues that are hard for the author and firstline reviewer to see because they are so close to the substance. “

Peterson has a chart that gives an overview to his proposed process that I am unable to copy and fit into PPT, so you’ll have to go look for yourself.

My quibble here is with Peterson’s bold font, second level, monitoring tradecaraft ” Is the piece consistent with previous analysis? “. There’s nothing wrong with that question, which is in fact a vital one to ask – the caution stems from the environment in which it is asked.

IC products are not the same thing as academic world, peer-review, though there are a lot of similarities because analysts are analysts. The question asked by Peterson very easily translates in the bureaucratic world to a driver to impose conformity on the new product in light of the position of internal vested interests who have authored the prior assessments. The IC misses new developments – the major paradigmatic ones – primarily because they are new and not part of the established pattern of experience that forms the IC conventional wisdom on a given subject ( in fairness, some ppl in the IC usually catch these new developments but their insights do not always survive the review to make it to intelligence consumers).

To maintain objectivity, at a minumum you would need to ask a few, similar, questions of the old IC products that preceded the new one being reviewed to try and minimize analytical bias. Questioning one’s own analytical perspective needs to become as second-nature as questioning the the argument and the accuracy of the data. To use a metaphor, if the IC’s model perspective on every subject is a magnifying glass then they ought to also get out a microscope, a telescope and use their naked eye once in a while.

NETWAR…4GW….NETWORK-CENTRIC…HELP?

If you are confused by some of the military strategic theory buzzwords tossed around here and on other blogs, Dan of tdaxp has a concise intro.

A GLITTERING EYE ON THE PENTAGON’S NEW MAP

Dave at the Glittering Eye has posted his official review of the Pentagon’s New Map and has solicited comments. An excerpt:

“I agree without qualifications with Barnett’s optimistic view of America’s grand strategy and with his unrelenting insistence that the benefits of the Lockean Core should be open to people of all countries regardless of gender, race, religion, or ethnicity. I agree with his view of the general benignity of America’s grand strategy although this view puts both of us at odds with a considerable fraction of the American electorate, mostly in the Democratic Party. I also agree with Barnett’s view of the relationship between the United States and China and that the likelihood of superpower war between the United States and China is extremely remote (I don’t necessarily agree with Barnett’s explanations for why this is so since I believe that issues completely internal to China are more significant than connectivity in explaining the relationship). However, I do think that Barnett’s model has some flaws and the remainder of this post will attempt to identify a few of them.

The first and most serious flaw is that Barnett never proposes a rigorous, quantifiable, testable definition of connectivity. The Core and Gap are areas on a map and Barnett tells us the difference is connectivity. The difference is that the area referred to as the Gap contains the problem spots and Barnett explains those problem spots by connectivity. We need a better test for determining whether a country is Core (which includes countries like the United States, France and Germany), New Core (which includes countries like China and India), Seam States (like Mexico, Brazil, and Turkey), or Gap (Iran, the Kingdom of Saudi Arabia, Nigeria, Colombia, etc.)….

…The second flaw is that I think that Barnett is a victim of his own metaphor. I don’t read Hobbes’s Leviathan as a call for a universal system of consensual rule by law. I read it as a demand for a powerful sovereign to keep the peace. When you characterize the U. S. role as Leviathan, it is in fact a call for an American Empire. Barnett is very clear in calling for American rules rather than American rule. But that’s not what Leviathan is about. He needs to come up with a different metaphor to avoid tearing down his own argument.

Barnett’s notion of dividing the responsibilities of the Pentagon between Leviathan responsibilities and what he refers to as System Administration responsibilities is ahistoric. Historically, several forces—genuinely different groups of people—have been influential in U. S. foreign policy decision making: isolationists, idealistic internationalists, economic realists, and populist nationalists. For a handy key to understanding these groups this post of mine.

Our military has historically been and currently is heavily influenced by Jacksonians. Jacksonians have little or no interest in the objectives Barnett is setting out for the SysAdmin force. That group will necessarily be composed of Hamiltonians and Wilsonians. You can’t carve out the function of SysAdmin. It will have to be added on both for political and druthers reasons. And I honestly don’t see any willingness of the American people to fund such a group as an add-on. “

You should read the whole thing. As usual, Dave is both measured and insightful in his criticisms, identifying several areas for additional questions or where Dr. Barnett’s thesis required greater support . Since Dave asked me for feedback, I have forwarded his post to Thomas Barnett and responded in Dave’s comments section, some of which I have reproduced below:

“* I too noticed that Core-Seam-Gap status could use some quick & dirty quantification. In one of my early PNM-related reviews I suggested a Bell Curve model be constructed. If PNM blooms into a full-blown analytical school of thought, as it has the potential for doing, that is an area for further work by somebody with a good grounding in quantitative analysis methodology.

* ” Leviathan ” – my read is that Dr. Barnett picked that nomenclature in the biblical-mythic sense of an awe-inspiring figure of titanic size and power and not the Hobbesian philosophical sense of the proper role of the State.

* Leviathan vs. System Administration forces: This would indeed require an unprecedented restructuring of American military foces that would dwarf the changes of 1947 when the Navy and War departments were fused into the DoD and the JCS and Air Force were created. The driver here is not Dr. Barnett’s book but the real-world pressure of repeated nation-building MOOTW deployments and the difficulties of the Iraqi insurgency.

* The need for the U.S. have some kind of force that specializes in ” nation-building” is becoming an inarguable consensus -what and who would compose a System Administration and how/when/where to deploy it is something hotly disputed. “

One of the nice things about PNM is that Dr. Barnett has to an extent, treated it as an ” open-system” to which others might make contributions or find new fields of application. Most theorists prefer to have ” closed-systems” in which their jealously-guarded work is considered a sacred text and their followers tend to sound fairly robotic – not to name any names like, say, Noam Chomsky, Immanuel Wallerstein, Edward Said….you get the idea.

What you have ultimately in the latter instance is a point of intellectual sterility when the theory, frozen in time, becomes less a model of reality than it does irrelevant dogma. That Tom has made the atypical, risk-accepting, choice at each stage of promoting his grand strategy is one reason PNM has continued to gain influence.

I’m already looking forward to reading Dave’s review of Blueprint for Action !

BRIEF ADVISORY

If you have been following the excoriating exchanges between Pundita and Collounsbury ( well,..Col was doing the rum-fueled excoriating, Pundita’s replies are mostly oblique sweetness and light- except for a brief burst of gigantic type font, since removed) these last few days, a commenter at Col’s referred to a site ” Antipundita” on blogger.

That site in question is a mock-up trolling for email addresses among other nasty things. It is my strong sugggestion that you do not send yours or even visit the site.

That is all.

WIDEN YOUR PARAMETERS

The U.S. Army War College journal Parameters is anything but a dull read. Foreign Affairs could take an editorial pointer or two from the Parameters staff in terms of selecting writers who won’t put their subscribers to sleep. Two winners from the current issue. The first article should be of particular interest to some of my more liberal fellow bloggers like praktike, Armchair Generalist and Mithras who check in here on occasion:

“Rescuing the Law of War: A Way Forward in an Era of Global Terrorism” by Michael H. Hoffman.

“The long-term import of recent trends can’t be overstated. The United States is surely—and not so slowly—bestowing legal status and privileges on members of terrorist organizations that have no precedent in the 3,500-year recorded history of warfare. Terrorists are acquiring legal recognition and support of a kind unavailable to members of US and other national armed forces, and for that matter unavailable to insurgents during civil conflict as well. (There are early intimations that the United States may end up unilaterally bestowing similar status and privileges on the members of opposing state forces as well as terrorist organizations.) The notion that opposing forces will ever make these unique legal privileges reciprocally available to the US armed forces simply doesn’t warrant serious consideration”

A counterintuitive opener but an analytically accurate one, as it describes a change to the operative interpretation to the Laws of War that the European Left have been pushing in diplomatic, legal and media circles since the 1970’s even as most EU states have effectively demilitarized.

The Euro-Left attempt to excuse unconventional fighters from Geneva Convention obligations and give them as ” edge” over pro-Western, conventional, government armies began during negotiations over the Geneva Protocols where our allies aligned themselves to a degree with the Soviet bloc on the issue. Back then, the guerillas were pro-Soviet Marxists fighting
” national wars of liberation”. Today they are Islamist terrorists with whose behavior gives them even less claim under Geneva to protection than the Marxist guerillas of yore. ” Lawfare”, so-called, is not a new phenomenon but an old campaign of the international Left.

The second article to which I’d like to draw your atention, has a great deal of continuity with the debate Cheryl Rofer and I recently had on History and Democracy promotion. Dr. Echevarria lays out a formidible case against oversetimating the uses of history ( or of historians, of which he himself is one):

” The Trouble with History” by Antulio J. Echevarria II

“The fundamental problem for historians is that, aside from being able to refer to such demonstrable facts as do exist, they have no objective references for determining (beyond a reasonable doubt) to what extent the histories they write either capture or deviate from the past. Put differently, they have nothing resembling the scientific method to aid them in determining whether what they have written is somewhat right, mostly right, or altogether wrong about the past. Quantitative history, intellectual history, “history from below,” and oral history, for example, each employ different methods. Yet none of those procedures can lay claim to the reliability of the scientific method—that is, developing a question or a hypothesis, conducting experiments to test it, revising the original hypothesis, then conducting further experiments to confirm the revised hypothesis, and finally reaching a conclusion.

To be sure, historians do have recourse to certain subjective measures—such as their abundant reviews of each other’s books and access to the advice of other, perhaps more accomplished, historians—to aid them in capturing the past. However, subjective measures tend merely to reinforce a veritable Cartesian circle of interpretation: historians write what they do based in part on the fragments of the past, but how they see those fragments is largely influenced by knowledge they have gained in the present, including the works of other historians who may indeed only be offering their best guesses as to what those fragments mean. This proved to be the case with historical interpretations of military thinking before the First World War; historians tended to view that era’s military theory and doctrine through a “lens colored red by the seemingly prolonged and futile slaughter of 1914-18,” and thus reinforced one another in a series of misunderstandings.

In addition, the impact of recent events or experiences sometimes causes historians to focus on factors and values that are quite different from what the historical actors had in mind—perhaps giving those factors and values an artificial existence. Hence, the present, as historian Christopher Bassford once noted, serves as “prologue” to the past. As Carl Becker explained, “Left to themselves, the facts do not speak. . . . [F]or all practical purposes there is no fact until someone affirms it.” And affirming a fact, of course, shapes how it is understood. Thus, historians tend to see in the past what they have been trained to see, or—for those inclined to buck convention (which requires a certain training of its own)—what they want to see. Neither tendency is necessarily wrong. Yet neither is necessarily right, either.”

A smart man. Historians do suffer from physics envy, perhaps not to the degree of some of the other social science disciplines but it exists because writing good history is fundamentally a craft. Less abstract than philosophy but also less factually concrete than geology, even the best researched historical narrative is going to have gaps and interpretation, putting oneself in the other’s shoes during the moment of causation or consequence, will always remain an educated guess.

The second problem highlighted by Echevarria can be addressed to a degree by deliberate introspection by the historian as to their epistemological perspective and forcing themselves to systematically look at the same data-set with different interpretive eyes. Hard to do but a worthwhile exercise in critical assessment. Echevarria himself gets very close to this suggestion when he refers to an old Education theory standby, Bloom’s Taxonomy:

” For their part, historians are after what Jack Hexter, one of the more famous and controversial of historians, once called that “elusive entity—the Truth.”They want to understand what really happened, whether or not it is actually possible to do so, and then to explain why it happened. Institutions of higher learning need professionals possessed of just such a “determination to find things out,” whether they succeed or not. Thus, the most valuable contribution that history and historians can make—and why they should remain integral to higher education—is that they attempt to understand things that lie outside the realm of certainty. Their answers may be flawed, but it would be unsatisfactory for the human species to limit itself to knowing only those things that can be verified by the scientific method.

Similarly, professional military education must equip students to understand the difference between historical reality (which, like the reality of the present, we may never fully know) and attempts to describe it. It must refrain from reinforcing the tendency among military students to regard history as, in Liddell Hart’s term, a “sentimental treasure.”42 Military professionals are better served by learning to be critical of the history that historians write, by building a habit of rigorously scrutinizing facts and sources, and of detecting biases and specious arguments, and by developing an eye for penetrating the myths that surround the past. They should regard the history they read, as Gaddis advises, as something between art and science. They must learn that a prerequisite to building a strong argument is the ability to recognize a weak one.”

If you blog on foreign or military affairs, Parameters is an indispensible read.