1

Critical Thinking for Strategic Intelligence

Instructors Materials

Chapter 1: Who Are the Customers? Review case study, Yemen: the Next Afghanistan? Does the situation continue to pose a serious threat to US interests?

Yes, AQAP is highly likely to remain a serious threat despite the departure of President Saleh. The death of al-Awlaki is a major milestone, but other leaders almost certainly will emerge to further his work. Moreover, a disintegrating economic and political situation in Yemen could create conditions highly favorable to AQAP to both recruit and train future terrorists. The Saleh regime supported US counterterrorism objectives for the most part and such support could become more questionable in the future.

Who would you consider the key customer in preparing this report? Most counterterrorism experts believe that Yemen could easily and quickly emerge as one of the most dangerous locations for hatching and launching terrorist acts against US interests and the West more broadly. For government analysts, the US President (or in other countries the Prime Minister or Chief of State) should be viewed as the principal customer for an analysis such as this.

o What would be the most important issues for the White House and the Presidents key counterterrorism advisors? Key issues for the President and the White House would be: How does the United States assess the current threat from Yemen? How might the threat evolve over the next few years (in both positive and negative ways)? Who and what dynamics could increase or diminish the potential threat?

o How would you define the key issue for the Secretary of State? Some key issues for the Secretary of State would be: How can the United States best support domestic and foreign pressure for reform and the move to more democratic institutions in Yemen? Who are the key partners? Who can exert the most effective influence in this process? How can the United States and the international community best mitigate potential problems created by a Yemen that becomes less stable in the future?

o How would you define the key issue for the Secretary of Defense? For the Secretary of Defense, some key issues would be: How do we sustain an effective counterinsurgency campaign in Yemen given all the economic and political uncertainties? Do alternative basing options need to be considered for US forces and force projection capabilities to effectively counter any threat posed by forces resident in Yemen? How would increased political instability in Yemen or the establishment of a new form of government affect the military balance of power in the region? What

2

could the US military do to influence these dynamics in ways that best support US interests?

Should the issue be posed as a threat, an opportunity, a key decision point, or some

combination of these? At this point in the story, the two issues most deserving of attention are whether Yemen poses a major threat and whether it offers the United Statesand the West more broadlya major opportunity for shaping a more positive future. Government analysts traditionally tend to focus much more heavily on threats, but policymakers always appreciate learning at the same time any insights analysts might offer regarding potential opportunities to shape the issue in ways that further national interests.

Are there critical unknowns, uncertainties, or contradictory information that should be highlighted? Several critical unknowns that require additional research or collection tasking include learning more about popular support for or grievances against the government, the potential role of tribal elites and Sunni Islamist extremists, the willingness of foreign powers to interveneand the willingness of humanitarian organizations to provide reliefif conditions deteriorate, and basic economic trends.

Chapter 2. What Are the Key Questions?

Review case study, Countering the Iranian Nuclear Threat: Stuxnet and Its Broader Implications.

What are the key questions a senior policymaker is likely to ask about the Stuxnet attack? Ensure that the responses satisfy the five characteristics of a good question. The Stuxnet attack almost certainly would prompt a number of questions ranging from the tactical to the strategic for any policymaker, including:

o Who is responsible for the attack? o How much damage was done? o Can the damage be repaired and how long will it take? o How far did it set back the Iranian nuclear program? o How are the Iranians likely to respond? o What implications might the attack have on the broader international legal system? o Were new precedents established regarding the future of cyber warfare?

What are the key questions those responsible for the security of industrial security systems

are likely to ask about the Stuxnet attack? Ensure that the responses satisfy the five characteristics of a good question. Officials responsible for industrial security most likely would ask the same set of tactical questions, but a much different set of operational questions, including:

3

o Were any of our systems damaged by the attack? If so, what is the extent of the damage and what has been done to correct any damage?

o Do the attackers pose any future threat to our systems? o Can the code be reconfigured to threaten any other systems? o Could we detect such an attack and what are we doing to prevent it from happening?

How would you use the Five Ws and H to identify the key question for a senior policymaker?

First, the analyst should generate the best answer to each of the Five Ws and H questions: o Who? The United States and Israel are cited as the most likely perpetrators of the

Stuxnet attack, but the possibility of another nation with sophisticated cyber capabilities or even a highly skilled hacker cannot be ruled out until more is known about the attack.

o What? Research undertaken in the last two years has removed most of the initial uncertainty. We now can say with a high level of confidence that the two digital bombs were released in a direct attack on Iranian nuclear facilities, and we know most of the defining elements of the attack.

o When? The attack commenced in July 2009 and continued to do damage through mid-2010.

o Why? The motivation for the attack almost certainly was to disrupt the Iranian nuclear program, focusing in particular on the processes for enriching uranium sufficiently to use it in nuclear weapons.

o Where? The two most obvious points of attack were the Bashehr nuclear reactor and the Natanz uranium enrichment facility. Other sites may also have been targeted, but we have no reporting of problems arising at other facilities.

o How? Subsequent research by computer experts has revealed that a virus was used to change a small piece of code in a component produced by the German company Siemens that the Iranians use to manage the operation of turbines at the Bushehr reactor and nuclear centrifuges at the Natanz facility.

For the senior policymaker, the answers to What? When? Where? and How? are fairly well-established, leaving Who? and Why? as the two questions most likely to demand their attention. For those responsible for industrial security, far more attention should be focused on the What? and the How?

How would you organize a paper on the implications of the Stuxnet attack using the Question Method? The first step would be to establish who is the key customer for the paper. Then a small but diverse group of analysts would be assembled to brainstorm a series of key questions the customer would be most likely to ask, probably drawn from the sets of questions provided above. The task of the group then would be to reorganize this list of questions from the most pressing to the least pressing from the perspective of the key customer. The paper would be organized around the final set of questions, addressing the most important question first and the question deemed to be least pressing last.

4

Chapter 3: What Is the Broader Context for the Analysis?

Review case study, Financial Crises in the United States: Chronic or Avoidable? Assume this case study was written in 2011 for generalists seeking to learn more about the US financial crisis that began in late 2007.

Describe the broader political and economic context when this article was written. How does this historical and theoretical background apply to the financial crisis of 2008? When this article was written in 2011, the United States was only starting to show the signs of recovering from a deep recession that had lasted for more than three years. US presidential elections would be held within the year. A major topic of conversation was what had caused the recession and what would be the best strategy for overcoming the suffering incurred since 2007. A major focus of the presidential campaign was how to create jobs while simultaneously taking definitive action to reduce a burgeoning federal deficit. Although only sporadic mention was made by the candidates of the various economic schools of thought, the positions espoused by the various schools were influential in framing the debate.

In the political discussions and debate surrounding the 2008 financial crisis, what was the context, and how was this portrayed? Much of the discussion in 2011 was focused on how to restart the economy or more specifically: how to create more jobs, how to reduce the federal deficit, whether this might include raising some taxes, and whether to consider major reductions in public entitlements. Some argued that the deregulation from the 1980s and 1990s was detrimental to the economyor at least inappropriately handledand called for increased regulation. Others countered that drastic action was needed to reduce the size of government and to eliminate many or most of the regulations facing the business community to stimulate more business activity and create jobs. They contended that care had to be taken not to strangle a faintly recovering economy. In fact, they believed that even more stimulus might be needed, justifying a short-term increase in federal spending. Such increased spending, in their view, could be offset by raising taxes on the most wealthy.

What are the key characteristics and strategies employed by the various economic schools of thought? Construct a matrix showing their most common criticisms of government, their preferred role for government, and the primary sources and proponents of their arguments. See Figure IV.1 Economic Schools of Thought: Key Characteristics and Strategies.

5

Figure IM.1 Economic Schools of Thought: Key Characteristics and Strategies

Economic School of Thought

Most Common Criticism of Government

Preferred Role of Government

Leading Proponents and Sources

Keynesian Economics

Too Little Regulation

Supports proactive government responses and reliance on fiscal stimulus policies during recessions

John Maynard Keynes, The General Theory of Employment, Interest and Money (1936).

Market Failure Theory

Inability to Overcome Market Inefficiencies

To intervene when free markets fail (such as with externalities) although this too can lead to government failure

Concept introduced by Victorian philosopher Henry Sidgwick and later developed by Herbert A. Simon, Hugh Gravelle, and Ray Rees.

Welfare Economics

Failure to Intervene to Provide Services

Assumes a benevolent and competent government that should be generous in providing social services and opportunities for its citizens

The early Neoclassical approach was developed by Edgeworth, Sidgwick, Marshall, and Pigou and the New Welfare Economics approach is based on the work of Pareto, Hicks, and Kaldor.

Chicago School/ Monetarism

Too Much Regulation

A laissez-faire policy and reliance on monetary policy to spur long-run economic growth

Milton Friedman, A Monetary History of the United States (1963).

Austrian Theory of Market Process

Distorting Prices

A laissez-faire approach driven by the purposeful actions of individuals and free markets

Writings of Carl Menger, Eugen von Bhm-Bawerk, and Friedrich von Wieser in late-19th and early-20th century Vienna.

List the key issues your customer (a well-informed, general reader) would like you to address in this article. Some of the key issues the general reader would like to see addressed are:

o Which positions of the various schools of economic thought best explain what caused the financial crises of the past few decades?

o Which positions of the various schools of economic thought best explain the ultimate policy outcomes of the financial crises of the 1980s? Who tells the best story?

o Which positions appear most relevant to address how to proceed in dealing with todays financial and economic challenges?

o To what extent was current policy fashioned by the philosophies of the various schools and which individuals were the most involved in propagating those views?

o To what extent are current policies a function of where policymakers come down with regard to the various schools of thought?

o To what extent do voter ideologies, positions, and interests drive policy? o Are the exogenous political variables cited in the case study ultimately more

influential than the philosophies expounded by the various schools?

6

How well did the author answer the so what question in the article?

The author provided substantial amounts of useful information relating to the so what question, but did not organize this paper around this question, opting to lay out the various arguments and let most of the facts speak for themselves.

Chapter 4: How Should I Conceptualize My Product?

Review case study, Yemen: the Next Afghanistan?

Assume that you are a US State Department intelligence analyst who has been asked to prepare an intelligence assessment focusing on the topic Yemen: the Next Afghanistan. Draw from the case study to perform the following tasks:

Write a sentence or two describing each of the AIMS of the paper.

Audience. The primary audience for this assessment would be the originator of US foreign policy in the State Department dealing with Yemen, such as the Assistant Secretary of State for the Middle East. In some cases, a more senior official might be deemed the primary recipient, such as the Undersecretary for Political Affairs, the Director of Policy & Planning, or the Secretary of State. Intelligence Question. The key intelligence question is whether the evolving dynamic in Yemen will force the United Statesand particularly the State Departmentto devote substantial and increasing resources and policy attention to the country in the coming year or two. Message. The message is that this could certainly prove to be the case if the succession process is not smooth and a vacuum of leadership begins to emerge, allowing terrorist elements to gain a stronger foothold in the country and threaten US interests. Story. The story is intended to take readers through a short history of political, economic, and cultural dynamics in the country to create a framework they can use for evaluating how events are most likely to unfold in the coming year or so.

Create a sample TOR. A sample TOR could include the following:

o The period of the estimate is two years. o Additional research will be needed on the behavior of key tribal leaders and other

officials likely to wield the most power after Presidents Salehs expected departure in 2012. Research will also have to be conducted to generate defensible economic projections for 2012 and 2013, particularly in the oil sector.

o Key information gaps include the likely growing role of tribal elites, popular support for political leaders, and the capability of social media to mobilize and articulate grievances.

7

o Strategies for filling these gaps include the submission of well-tailored intelligence collection tasking as well as debriefings of former State Department officials who worked closely with Yemenis in the past.

o A colleague who is an economist will be tasked with conducting the economic trends analysis.

o A professor who is knowledgeable on the topic and has delivered useful papers in the past will be asked to write a ten-page paper discussing likely directions in tribal politics and who is likely to emerge as a key power broker.

o The analysis would benefit from three techniques: a Key Assumptions Check, a Multiple Scenarios Generation exercise that focuses heavily on identifying the most critical key drivers, and the development of a set of validated and diagnostic indicators for tracking future developments and best anticipating how Yemen will actually evolve.

o A timeline will be established with the following target dates: Key Assumptions Check in 1 week; Research and interview completed in 3 weeks; economic contribution in 3.5 weeks; Multiple Scenarios Generation and Indicators workshop in 4 weeks; first draft in 6 weeks; complete coordination in 7 weeks; final draft to customer in 2 months.

Review the Getting Started Checklist (Figure 4.2) and identify five things you need to focus

on before beginning to draft the assessment. 1. (Question 2). What is the key concern regarding Afghanistan that we need to address in

the Yemeni paper? Key concerns include the role of tribal politics, the impact of the drug culture, the implications of weak central government, and/or the difficulty of extracting a super power from the situation after it becomes heavily engaged.

2. (Question 5) Do I need to establish exactly for whom I am writing this paper? If for the Assistant Secretary, then I will not need to provide a lot of background information; if for the Secretary, then a lot more attention will have to be paid to tribal and cultural issues and past political maneuverings.

3. (Questions 7, 8 & 11). This topic will not likely have a single answer. Consequently, we should give serious consideration to generating several possible scenarios and focusing the paper on key drivers of future events. A Key Assumptions Check would also be very useful as we kick off the project and help us challenge whether there are true parallels between Yemen and Afghanistan.

4. (Question 9). A lot of the critical background information relates to internal tribal politics. We usually do not get good reporting on this topic. This suggests it would be prudent to identify some professors with richer knowledge of this topic who can contribute to the product.

5. (Question 10). We should identify and reach out immediately to some Yemeni as well as Afghan experts, including former political and consular officers and perhaps former ambassadors who could provide valuable insights and perspective on this topic. Their knowledge and insights will help focus research and surface new issues and key drivers we have yet to consider.

8

Chapter 5. What Is My Analytic Approach?

Review case study, The End of the Era of Aircraft Carriers. Briefly answer the following questions:

How would you define the issue for a global strategist, naval commander, weapons designer, or a senior policy official?

Key customers are likely to view the issue from different perspectives, reflecting what is most important in their particular job. For example:

o A global strategist would focus on what impact the demise of the aircraft carrier as a platform to project national power will have on interstate relations and interstate conflict. What new instruments such as cyber might fill this vacuum?

o A naval commander would want to test the assertions that a carrier strike group is increasingly vulnerable and explore whether new technologies could reverse this problem. He or she might ask what will be the priority missions of the carrier battle group of the future.

o A weapons designer would be most interested in learning how the demand for offensive and defensive weapons technologies will evolve over the next ten years. How will the nature of conflict change and what types of new weapons need to be developed?

o A senior policy official would have a shorter time horizon, focusing on how quickly this shift is likely to occur, what implications it has for foreign relations with key global partners and adversaries, and how a vision of a more cooperative maritime future could best be achieved.

What information is each customer likely to request?

Key customers will have different requirements, reflecting what is needed to do their jobs. For example:

o A global strategist would want more information on the economies of nations with aircraft carriers and expert projections on how soon they may be forced to decommission the ships. He or she might also want to know whether military strategists in the countries mentioned in the case study are already adjusting their plans and procurements in response to these drivers.

o A naval commander would want more information on anti-ship missile, torpedo, and drone capabilities; more data on the prospects for developing better defensive capabilities; and estimates of how many countries are likely to possess the weapons needed to deter or defeat a carrier battle group.

o A weapons designer would be most interested in more specifics on current and projected capabilities of both the offensive and defensive weapons systems discussed in this case study. Another issue would be whether sufficient funding will be available to support the development of new weapons systems.

o A senior policy official would want to know how national leaders in other countries are dealing with this phenomenon and what expectations are being generated about the future role of the United States in projecting a global naval presence.

9

Where would you look for information on the new technologies and evolving naval strategies? A major source of such information would be popular journals and other periodicals in the areas of military strategy and doctrine, weapons development and procurement, emerging technologies, and economic projections on a global and national scale. Other sources include Internet sites that specialize in naval capabilities and weapons, testimony before the US Congress, and foreign government publications on naval strategy.

Is your analytic argument descriptive, explanatory, evaluative, or estimative? Does sufficient data exist to address this issue? Although this case study falls into the category of estimative analysis, elements of the three other types of argument are also reflected in the study. For example, the data provided in the chart on global naval resources is descriptive and the discussion of why the United States is currently unchallenged as a naval super power is explanatory. The sections on the shrinking global naval presence and the three key drivers contain both evaluative and estimative analysis.

How could a conference of experts help to answer the key questions? Much of the analysis in this case study is speculative. A conference of experts would provide an efficient mechanism for validating some of the trends presented. Assembling a group of experts from a variety of disciplines, including naval strategy, weapons developments, technology, economics, and foreign relations, could stimulate a rich discussion of the key drivers and likely outcomes by forcing most participants to think outside their traditional box. Such a group of experts would help bound the issue in terms of the seriousness of current vulnerabilities and how quickly new and more threatening weapons technologies are likely to emerge. Ideally, the experts would pool their expertise to construct several credible scenarios of how this issue will evolve, providing policymakers with a firmer baseline for future planning.

Chapter 6. Can Collaboration Contribute to a Better Answer?

Review case study, The End of the Era of Aircraft Carriers.

The article involves analysis of military, political, economic, and technological topics. The author was an expert in only one of these areas. What would be a good strategy to get the authors peers in the other disciplines to collaborate in producing the article? The most important strategies focus on the beginning stages of drafting. A critical first step would be for the author to alert his or her colleagues who work on different aspects of the topic of the authors intent to write a paper. Colleagues might contribute information or even papers they wrote that touch on key aspects, saving the author time in drafting. They might also point the author to key sources of information or other papers that relate to the topic, once again saving research time. More important, the initial contacts would reveal if colleagues have already formed strong views on the topic that could differ from those of the

10

author. The question then becomes what information and insight helped to form these opinions and how credible is this contrary evidence.

Another way to spur greater collaboration would be to invite colleagues to a concept paper session or a Terms of Reference meeting. The author would benefit from any good ideas that emerge from the discussions and might even persuade colleagues to contribute sections to the paper, thus capturing their expertise and saving drafting time. Colleagues should be given advance notice when a draft will be distributed for coordination to ensure they have sufficient time to ask probing questions and provide useful commentary. A good critical thinker will plan to have sufficient time to process the comments of his or her colleagues and recraft the paper accordingly. Attribution and credit for any contributions should be provided in accordance with office practice.

When considering the six imperatives, which would you expect to offer the greatest

opportunities and which the greatest challenges? The article makes a bold statement that will almost certainly capture the attention of senior officials. The drafting team would almost certainly share a sense of mission criticality and would mutually benefit from working together in representing different analytic disciplines and possibly different organizational cultures. If the author adopts a collaborative approach to drafting the paper as outlined in the previous question, this should assist in developing mutual trust.

Access and agility could pose a challenge depending on where the collaborating team members are located and whether the existing IT systems support collaborative endeavors. Unfortunately, collaboration usually works well when teammates work in the same office or building, but can prove challenging if separated geographically, especially if over several time zones.

Taking time at the start of a project to establish a common understanding could prove particularly importantand challengingin drafting this paper given the non-traditional approach. In the best of circumstances, the authors superiors would assume that the need to offer incentives falls mostly on their shoulders. The managers of the author and the contributors could consult at the beginning of the project to determine how best to ensure that all team members get credit for contributing to the final product. The author might also offer incentives in the form of agreeing to provide input to papers contributors plan to write or to volunteer to provide peer review for their future papers.

If the analyst had access to a collaboration cell, how might the cell have assisted him or her?

A collaboration cell could assist the author by creating a collaborative network for the drafting team or establishing a wiki or blog to facilitate collaboration. Cell members might also assist the author and the team in applying the What If? Analysis methodology or other structured techniques like Structured Brainstorming or a Key Assumptions Check, if appropriate. If major gaps exist in the information or expertise, collaboration cell members might be able to locate the needed expertise elsewhere within or outside the organization and facilitate contact.

11

If conflict arises in coordinating this article within the authors organization, would you expect the arguments to center on different facts, different interpretations, or different goals and objectives?

o For this mostly speculative topic, differences of fact probably would be resolved without much conflict by reviewing the credibility of sources and the presence of other facts that provide independent confirmation. Analysts often work from different data sets, and they need to share their entire data base with their peers. The challenge is to identify early on when and where analysts are working from different data bases or their facts are in conflict.

o Most of the conflict is likely to revolve around differences of interpretation. Analysts from different functional disciplines process data in different ways; this could impede dialogue.

o Serious differences could emerge, however, over goals and objectives because the paper posits change that would be a major departure from several decades of military practice. Naval or military analysts could be deeply engrained in mindsets that support the value of surface naval dominance and might resist the argument that both technology and economics will force a major reassessment of the costs and benefits of aircraft carrier battle groups.

Chapter 7: How Do Models Help My Analysis? Review case study, Financial Crises in the United States: Chronic or Avoidable? Assume this case study was written in 2011 for generalists seeking to learn more about the US financial crisis in 2008.

What models did the author discuss in the paper? How did this affect the politics and politicization of the problem? The case study provided an overview of the prominent economic schools of thought including the Keynesians, Chicago School/Monetarists, Austrians, Market Failure Theory, Welfare Economics, and the Public Choice/Political Economy. Presenting a broad spectrum of economic and political thought helped establish an objective baseline for efforts later in the paper to evaluate the impact of each school and which were the most influential.

Figure IV.1 The Financial Crisis: Economic Schools of Thought represents another example of how the influence of economic schools of thought on decision makers can be modeled in a visual format. In this instance, the author sketched a new model of how economic schools of thought influence the economic decision making process and frame interpretations of history. The model suggests these dynamics could also be self-reinforcing.

Would you say these models were drawn from Traditional Analysis, Structured Analysis,

Empirical Analysis, or Quasi-Quantitative Analysis? These models were drawn from a combination of traditional analysis, empirical analysis, and quasi-quantitative analysis. The schools of thought were developed before the emergence of structured analytic techniques as a form of analysis.

12

o Traditional/Structured Analysis: There is a definite qualitative analysis component to all schools of thought. In fact, the Austrian tradition claims to be an extension and elaboration of Adam Smiths works. While Smith is commonly known as the first economist, he was actually a political philosophereconomics only became a separate field much later. Smith (and later Hayek, Mises, other Austrians, Keynes, and Chicago School economists) have derived much of their philosophies from qualitatively analyzing history and drawing conclusions and policy recommendations.

o Empirical Analysis: Chicago School economists/Monetarists fall more into the realm

of quantitative analysis. For example, they focus on questions such as how much to contract or expand the money supply, and which method is most appropriateboth largely empirical questions. Keynesians can be quantitative in their estimates of stimulus package requirements. Both schools use empirical data to analyze and justify their policy decisions or, if their policies were not implemented fully, argue why the policies failed.

o Quasi-Quantitative Analysis: This form of analysis is mostly associated with the

Austrian school. Generally Austrian economists are not very quantitative but will systemically analyze patterns to assess tendency and direction of an economy, based on various indicators. They may perform some structured analysis, but they are not explicit about using structured analytic techniques.

How did the use of the models facilitate your understanding of the issue?

The review of the various schools of thought and the key tenets adhered to by each school helped the reader understand both the complexity of the issue and the underpinnings of the political debate within the general public about which policies are likely to prove most effective in improving economic performance in the United States and the world. Many readers probably recognized the teachings of the Keynesian and Chicago School theorists but were less familiar with the positions of the Austrians and other schools. Given the growing influence of the Austrian and Public Choice schools, the case study helped put their views in perspective. The use of a summary graphic to represent key streams of thought and their impact on economic decision making also helped to clarify some issues and provide a context or framework for discussing and debating the impact of the various schools.

Did the use of models suggest any key information gaps?

The economic frameworks focused on key economic decision making processes but gave short shrift to politics and made no mention of social and demographic trends. The author did not explore the parallel economic and financial crisis in Europe and did not touch on how other international players (China, Brazil, India, the Middle East oil producers) could impact the system. Another key information gap is the lack of a concluding discussion of what current data says about whose interpretation of history is correct.

13

Chapter 8: What Types of Information Are Available? Review case study, Puzzling Food Poisonings in Germany.

Where should we look to find authoritative information about the food poisoning outbreak reported in Germany? In cases such as this, a plethora of information can usually be found on the Internet. Unfortunately, much of that information is speculation or the repackaging of information published by other sources, mostly newspapers, wire services, and 24-hour television news channels. Separating the wheat from the chaff can prove a major challenge. The key is to trace all press reporting back to an original source and then evaluate the credibility of that source before entering it into your database or using it in your paper. A more efficient way to collect critical data is to identify government press spokespersons or official government web sites that are responsible for informing the public about the outbreak. They are often more comprehensive in their reporting and are less likely to continue to repeat information that has already been released.

As you start your research, what is known and what additional information is needed? A careful review of the available information at the start of this case study reveals that little solid information is available except that growing numbers of people are becoming ill from an E. coli bacteria. Most of the reporting focuses on individuals in northern Germany, but the absence of reports from other geographic locations does not necessarily mean that the outbreak is isolated to one area. In fact, in this case study similar outbreaks were reported in North America and France, and the disease could have easily spread to many other locations given the high volumes of cross-border traffic. Key unknowns include the food source that is responsible for the poisoning, the origin of that food source, the type of strain, how easily one could become infected, and how lethal the bacteria were. While most of the press reporting was focused on what food was responsible for making people sick, over the long run answers to the other questions were far more important to track down.

Another caution is that, even assuming honesty on the part of the information sources, some had good reason to withhold information. German officials, in fact, were criticized for speaking too soon. Experts initially had thought the source of the outbreak was Spanish cucumbers, but had not verified this before publicly announcing it. The announcement unnecessarily caused millions of Euros in damage to Spanish farmers. As a result, governments and other sources of expertise are likely to be even more reluctant to provide information in the future.

What are examples of primary and secondary sources used in this case study? Primary sources used in the study include Professor Weidmann, who was interviewed by Time, an official WHO report, German Health Minister Bahr quoted by BBC, WHO official Flemming Scheutz quoted by Nature, and the statement by the European Food Safety Authority reported by the Guardian. All the other sources qualify as secondary sources in that they summarize what other people have said or reported. All of this information was derived from reputable news sources, but much was still conflicting, confusing, and in the case of the Spanish cucumbers inaccurate.

14

What tangible evidence is used in this case study? Is the case based mostly on tangible or testimonial evidence? Which is the more trustworthy? Examples of tangible evidence include official WHO and other European documents as well as the forensic evidence reported by German scientific organizations and the Chinese institute that identified the genome. The preponderance of the reporting, however, was testimonial evidence in the form of press interviews of government officials, knowledgeable academics, and ordinary citizens. Caution should be taken in using much of the press reporting, the bulk of which is second and third-hand.

Chapter 9: Can I Trust the Sources? For this chapter two case studies have been used: Puzzling Food Poisonings in Germany for assessing the quality of sources and Countering the Iranian Nuclear Threat: Stuxnet and its Broader Implications to explore the potential for deception. Review the case study Puzzling Food Poisonings in Germany to examine the quality of the sourcing. What demonstrable tangible evidence is presented in this case? How would you rate its

credibility? Examples of demonstrable tangible evidence include the WHO Global Alert and Response Report posted on the WHO website and the Reports of the Robert Koch Institute, Germanys national disease control agency. Both sources rate High in all three categories: authenticity, the reliability of the sensing mechanism, and the accuracy of the representation. Key findings contained sufficient detail, and the sources have well-established reputations for accurate reporting. Nevertheless, caution should be observed in situations of this kind because government officials are under extreme pressure to tell the public something even if it has not been completely verified. German officials fell into this trap when they first reported that Spanish cucumbers might be the origin of the infections.

Several individuals and institutions are quoted in this case study. How would you rate their competence and credibility? All the sources quoted in this case study received relatively high ratings (see Figure IM.2). Many other sources could have been included in the paper that would have received much lower ratings, particularly those that involved individual or press speculation about the origin of the disease.

Are there any examples where the credibility of the collector could be confused with the credibility of the source? Two examples where the credibility of the collector could be confused with the credibility of the source include early reporting from CNN that the cause of the outbreak was cucumbers from Spain and press reporting from Hamburg that Spanish produce was to blame.

15

Figure IM.2 Credibility of Human Source Reporting on German Food Poisoning Event

Source: Randolph Pherson, Pherson Associates, LLC Were any sub-sources used whose credibility was not established?

Some of the press reporting involved sub-sources whose identity was not established, preventing readers from evaluating the credibility of the sub-sources. The best example of this was the article from the Sky Valley Chronicle in Washington State that provided no information on any original sources used by the reporter.

Review case study, Countering the Iranian Nuclear Threat: Stuxnet and Its Broader Implications, to explore the issue of deception.

Which of the six reasons for being concerned about deception might be relevant to the story of the Iranian Stuxnet attack? In the early stages of the attack, the primary source of information was the Iranian government, which would be highly motivated to deceive and well practiced in conducting deception operations. The Iranians also would stand to lose a lot if they were truthful because any admission of damage would offer the attackers valuable feedback on the success of the operation and insight on how best to stage additional attacks. From the perspective of Iranian decision makers, the ideal strategy would be to convince the perpetrators that the attacks did little damage and the costs of mounting such a sophisticated attack greatly outweighed the benefits.

Would the Iranians have reason to deceive other countries and the scientific community about the extent of damage done by the Stuxnet virus on their systems and programs? Why? How would this be reflected in the reporting? The Iranian government would have every reason to deceive the world about the impact of the Stuxnet virus. It would not want adversaries or global public opinion to believe its nuclear systems were vulnerable to such attacks; it also would not want likely perpetrators to

Competence Credibility

Access Expertise Objectivity Veracity Observa-tional Sensitivity

Cultural Perspective

Professor Weidmann, Cornell University Medium High High High Medium Medium

WHO Global Alert & Response Report High High High High High High

German Health Minister Bahr High High Medium High Medium High

WHO official Flemming Scheutz High High High High High High

European Food Safety Authority Report

High High High High High High

16

learn how effective the attack actually was. For these reasons, official Iranian reporting should be expected to deny or downplay the impact of the attacks.

Would Israel and the United States have reason to deceive the world about whether either country was involved in propagating the Stuxnet virus? Why? How would this be reflected in the reporting? Israel and the United States would be highly motivated to deny any involvement in the Stuxnet attack. One of the biggest challenges in the cyber realm is to identify the actual source of malicious code. This makes plausible denial an easy standard to attain. Moreover, admitting responsibility for the attack could inflame Iranian public opinion, provide the Iranian government with justification to retaliate, and potentially violate international law.

What questions should analysts have asked when first receiving reports that a computer virus had infected the cockpits of US Predator and Reaper drones? In what way could one argue that such reports constitute deception? Several questions are directly relevant and raise the possibilityin the early stagesthat either a deception operation was being conducted or analysts were engaging in self-deception. The reports linking the virus attack on the drones were speculation based on blog reporting on the Internet that spread rapidly across the cyber community. This would give a potential deceiver plausible deniability largely because the actual initial source of the report would be difficult to determine. What made the speculation compelling is the Iranians would have motive to retaliate, giving the initial reports a germ of credibility. Moreover, the Iranians have a history of being deceptive and much to gain or lose if the drone program is cancelled or suspended because it is deemed too vulnerable to cyber attack. Lastly, the Iranians could easily track how the US military was reacting to the reports given the amount of press speculation spawned by the initial reports.

Which of the seven rules of the road would be most useful in assessing whether the drone virus reports might be inaccurate or deceptive? Key rules to consider would include:

o Rule Two: Seek the opinion of those closest to the attack. In fact, once military authorities conducted their own investigation of the unit and its computers, the reported facts of the case changed dramatically and speculation of an Iranian source proved unfounded.

o Rule Three is highly applicable because no tangible proof exists. The reporting either was acknowledged speculation or emanated from Internet sources that could not be verified.

o Rule Four: Similarly, the Internet usually qualifies as verbal intelligence in that the veracity of what is stated in a message can be extremely difficult or impossible to validate.

o Rule Seven correctly prompts the analyst to avoid falling into the trap of satisficing. In this case, the Iranians would have motive and might have capability; lacking a serious examination of alternative explanations, the analyst should avoid jumping to the conclusion that the Iranians were responsible.

17

Chapter 10. How Should I Evaluate Sources on the Internet? Review case study, Countering the Iranian Nuclear Threat: Stuxnet and Its Broader Implications. Which Internet sources used in this case study should be investigated more closely?

Several reports quote journalists from well-respected publications or television networkssuch as The New York Times, CNN, and ABCand can be assumed to be reliable. Haaretz.com, Wired, and IDB News/PCWorld Business Center are less well known; prudence suggests probing the track record of the named reporters as a double check. Additional Internet research reveals that all three sources appear reputable.

o Yossi Melman is a senior commentator for the Israeli daily Haaretz. He specializes in intelligence, security, terrorism, and strategic issues and has written seven books on these topics, including The Nuclear Sphinx of Tehran: Mahmoud Ahmadinejad and the State of Iran.

o Noah Shachtman is a contributing editor at Wired magazine and editor of its national security blog, "Danger Room." He has reported from Afghanistan, Israel, Iraq, Qatar, Kuwait, the Pentagon, Los Alamos, and from military bases around the United States. He has written about technology and national security for The New York Times Magazine, Slate, Salon, Esquire, Popular Science, The New York Post, Foreign Policy, Popular Mechanics, The American Prospect Online, The Forward, and The Bulletin of the Atomic Scientists, among others.1

o Robert McMillan is a senior writer for IDG News Service. IDG stands for the International Data Group, which claims it is the worlds leading technology media, events, and research company with media brands including Computerworld, Macworld, Network World, PCWorld, and TechWorld that reach an audience of more than 280 million technology buyers in 97 countries.2

The remaining source is the blog by Tyler Durden on Zero Hedge. An investigation into this source raises some serious questions. Zero Hedge is a financial blog founded in January 2009 that claims 333,000 visitors a month, mostly from Wall Street.3 It was founded by Daniel Ivandjiiski, who claims he writes for the blog as part of a group of writers that uses the pseudonym Tyler Durden, a character in the movie Fight Club. Ivandjiiski is a 30-year-old Bulgarian immigrant banned from working in the brokerage business for insider trading. A former hedge-fund analyst, he is also a zealous believer in a sweeping conspiracy theories.4 Zero Hedge is best known for alleging that Goldman Sachs was using sophisticated, high-speed computers to siphon hundreds of millions of dollars in illegitimate trading profits from the New York Stock Exchange, which subsequently led the US Security and Exchange Commission to consider banning that form of computerized trading.5 Given the anonymity of

1 Accessed on December 11, 2011 at http://www.noahshachtman.com/about.html 2 Accessed on December 11, 2011 at http://www.networkworld.com/Home/rmcmillan.html 3 Damien Hoffman, "First Amendment Award for Outstanding Journalism: Best Blog Zero Hedge" Wall St. Cheat Sheet (August 4, 2009). Includes interview with co-founder "Tyler Durden." Accessed on December 11, 2011 4 Joe Hagan, "The Dow Zero Insurgency" New York magazine (September 27, 2009). Retrieved December 11, 2011 5 Joe Hagan, "The Dow Zero Insurgency" New York magazine (September 27, 2009). Retrieved December 11, 2011

18

the source and the maverick reputation of the blog, additional research would be warranted to confirm the allegations made in the reporting.

Complete the checklist shown in Figure 10.2 for the most suspect Internet source.

See Figure IM.3.

Figure IM.3 Four Step Web Page Evaluation Checklist: zerohedge.com

Name of Website: Zero Hedge URL / Address: http:// www.zerohedge.com 1. Type Advocacy x Business Info/Ref x News

Personal Entertainment Meta-tags Who are they trying to attract to their website (view / source)? Keywords?

Older Versions How did the site evolve? (check archive.org)

Founded in January 2009 by Daniel Ivandjiiski, a former hedge-fund analyst banned from working in the brokerage business for insider trading. According to archive.org, it has been crawled 260 times going back to February 13, 2009.

2. Content How reliable is the information?

Reliable Cant say x Unreliable Accuracy - Errors of fact or logic - Misspellings, poor grammar - Incorrect dates

Several articles from reputable journals state that the blog is known for promoting conspiracy theories. At times, however, its muckraking can reveals true problems or vulnerabilities in the financial system such as its revelations about inappropriate flash trading at Goldman Sachs.

Authority - Author unqualified, uncited - Poor reputation - Sources undocumented

Author is anonymous. A group of unknown individuals write the blog under the pseudonym of Tyler Durden, a character in the movie Fight Club. The blog claims anonymity is required to protect contributors from retaliation from the targets of their investigative reports.

Objectivity - Any blatant bias (terms, etc)? - Persuasive aim? - Single or multiple Point of View? - Any sponsors or advertising?

The founder of the website was banned for insider trading and obviously has a grudge against Wall Street. He advocates a total crash of the stock market that would lead to the elimination of the big banks altogether and the reinstatement of genuine free-market capitalism.

Currency - Out-of date references - When was it last updated? - Any dead links? (also check www.brokenlinkcheck.com)

www.brokenlinkcheck.com was used to check 1660 web pages revealing only 20 broken or dead links, suggesting the site is active and well maintained.

19

Coverage - Any significant omissions?

Reporting is highly biases and lacking of presentations of balancing perspectives.

3. Owner / Author: Full legal company name: -Check links to copyright and privacy statements.

ABC Media, Ltd

Who registered the domain?

The domain is registered to ABC Media, Ltd. in Sophia, Bulgaria. No suspicious information was noted.

Who incorporated the company? Officers / directors?

The administrative and technical contact is listed as: Hristozov, Georgi abcmedialtd@hotmail.com P.O. Box 814, Sofia, Sofia 1000, BG Hristozov is described by LinkedIn as a software developer at the Bulgarian Academy of Sciences

4. Affiliations and Associations? Who do they link to? Shared premises? Google Phone numbers and addresses.

Zero Hedge website has extensive links and postings to other blogs, websites and articles covering global financial issues.

Who links to them? - Nature of association - Effect on credibility

Zero Hedge website has extensive links and postings from other blogs, websites and articles covering global financial issues.

What do others say about them? Google names. Check for urban legend, hoax, or fraud.

Snopes.com reports no hits for zerohedge.com. The fraud.com and webglimpse.net search sites revealed no hits.

Is it consistent with similar sites?

Yes

Do any of the exceptions to traditional sourcing rules apply to the sourcing used in this case study? No.

How likely is it that the answer to the following question can be found on the Internet:

Did Iran conduct a virus attack on US drones in retaliation for Stuxnet? The answer to that question cannot be found on the Internet. At best, one might be able to find speculation that Iran was responsible for the attack, but the chances of someone providing actual evidence that Iran was responsible is minuscule. The only authoritative source would be a statement by an Iranian official that Iran conducted the operation accompanied by concrete evidence proving that it launched the attacka standard that is almost certainly never likely to be met given Iranian concerns that the United States could use such a statement to justify retaliation. The best answer to that question lies in constructing an argument laying out the pros and cons and systematically evaluating the evidence and the logic that would support such an assertion. In addition, the argument should demonstrate that any alternative explanations are less likely to be true.

20

Chapter 11. Are My Key Assumptions Well-Founded? Review case study, Puzzling Food Poisonings in Germany.

At the outset of this case, what key assumptions were reporters (and analysts) making about the outbreak of E. coli in Germany? How many were supported, in need of caveats, and unsupported? A summary of the key assumptions is provided in Figure IM.4. In this exercise, four of the assumptions are supported, two are caveated, and four are unsupported. The caveats are that 1) the bacteria is contagious only if direct physical contact is made, and 2) initial research suggests that the strain is uncommon, but additional research is required to determine the exact strain and its origin.

Figure IM.4 German Food Poisoning Key Assumptions Check

Number Key Assumption Rating 1. The strain of bacteria was more toxic than most. S 2. The food source of infection was cucumbers. U 3. The food source of the infection was raw vegetables. U 4. The geographic source of the infection was Spain. U 5. The geographic source of the infection was northern Germany U 6. Official reports citing the likely source of contamination would

cause serious economic damage to farmers. S

7. The strain has never before been seen before in humans. C 8. The strain was highly contagious. C 9. Victims could be treated with antibiotics. S

10. Washing ones hands and cooking all vegetables would reduce the chances of infection. S

Legend: S = Supported C = With Caveats U = Unsupported

Were one in four key assumptions unsupported as is usually the case? In this example, the ratio of Unsupported to Caveated and Supported assumptions is higher than the average of one in four. In this case, four out of ten assumptions are Unsupported. This is not surprising given the lack of solid information at the onset of the crisis and public pressure to provide useful data people can use to avoid becoming infected.

How would a Key Assumptions Check exercise helped German officials working this case? If analysts supporting German officialsand reporters for that matterhad conducted a Key Assumptions Check as the breakout was in its early stages, the chances of suggesting incorrect sources of infection probably would have been reduced.

21

A reasonable assumption initially made by the experts was that the strain of bacteria originated in animals. This assumption was reasonable because most E. coli strains that lead to outbreaks originate from animals, which is transmitted to humans through food and water. Officials were quick to test this assumption in the lab, however, and it was fairly quickly found to be untrue. As a result, the scientists concluded that humans were the most likely source of the outbreak, even if food was the way in which it was transmitted.

If a Key Assumptions Check were conducted, what key uncertainties would have been identified suggesting priority topics for investigation? A Key Assumptions Check would have identified several obvious areas for investigation, including what food source was causing the problem and where the source originated. More importantly, a Key Assumptions Check might have prompted officialsand reportersto delve more deeply into what strain of virus was causing the outbreak, how contagious and lethal it was, what would be the most effective treatment, and whether more care should be taken before announcing possible sources of the bacteria.

Chapter 12. Can I Make My Case? Review case study, Countering the Iranian Nuclear Threat: Stuxnet and Its Broader Implications.

What is the primary assertion made by the analyst in this case study? What key evidence and logic does the analyst provide to support this claim? The primary assertion made in the article is that a computer worm, dubbed the Stuxnet virus, was inserted into Iranian computers to sabotage Irans nuclear program. The motive for such an attack is well established in the first page of the article that discusses international concern that Iran will soon acquire the capability to make a nuclear weapon. The process by which the worm was inserted into Iranian computer systems is well documented by several sources as is the impact on the nuclear program.

What are two key secondary assertions and the key evidence and logic used to support the

claims? Key secondary assertions include:

o The Stuxnet virus did substantial damage to the Bushehr nuclear reactor and the Natanz uranium enrichment facility.

o The United States and/or Israel are responsible for launching the Stuxnet virus. o Iran retaliated by infecting the computer controls of US Predator and Reaper drones.

Substantial evidence is provided to support the first claim, weaker evidence to support the second, and no evidence to support the last. The case study uses several independent sources, including reports from the Iranian government to document both directly and indirectly the amount of damage done by the worm to both the Bushehr nuclear reactor and the Natanz uranium enrichment facility. The assertion that the United States and/or Israel were responsible for the attack is supported by logic (public statements made by leaders of both countries that Iran should not become a nuclear state) and unconfirmed press reporting from

22

unnamed sources that Israel had built a test facility and Siemens had participated in a program in Idaho to identify vulnerabilities in computer controllers identical to those used by the Iranians. No specific evidence is provided, however, that the US or Israeli governments were involved in implementing the actual attack. The third claim that Iran retaliated by infecting the control systems of US drones is pure speculation, later disputed by US Department of Defense press statements.

Are any contrary views or evidence presented that would challenge the key assertion or the secondary assertions that you identified? The case study presents the views of Iranian officials that challenge the contention that Iran is intent on developing a nuclear weapons capability as well as early reports that a virus had seriously damaged some of their nuclear programs. The study also presents an alternative hypothesis that much of the damage could have resulted from the purchase of flawed equipment provided by Western intelligence agencies through fake companies. In the case of the virus in the Predator control systems, a credible alternative explanation is provided by a US Air Force spokesperson.

Can you find examples of each type of analysisdescriptive, explanatory, evaluative, and estimativeas it appears in the case study? The technical discussion of how the virus most likely was inserted into the Iranian computers and the way it operated is a good example of Descriptive Analysis. The exploration of whether the United States and/or Israel were responsible for the attack is a good example of Explanatory Analysis as it tries to link several distinct events to historical trends as well as US and Israeli motives for launching such an attack. An example of Evaluative Analysis would be the discussion of factors or criteria relating to the broader concern that someone could customize the open source worm to launch potentially more potent attacks in the future. The last two paragraphs of the case study pose several challenging questions that would require a response using Estimative Analysis: How will cyber attacks be distinguished from other forms of conventional attack in the future? Will new international norms be established and how would they be enforced?

Can you find any examples of logical fallacies in the case study? The assertion by the Zero Hedge blogger that Iran may have been infected the Predator control systems in retaliation for the Stuxnet attack is a good example of several common fallacies, including Hasty Generalization and Post Hoc, Ergo Propter Hoc. The Stuxnet attack had attracted a lot of attention in the cyber world. When another report surfaced of a possible cyber intrusion of US systems, a Hasty Generalization emerged suggesting that it was a retaliatory attack. The fact that the Predator virus occurred after the Stuxnet attack is not sufficient to argue that the first event caused the subsequent event to occur.

23

Chapter 13. Did I Consider Alternative Hypotheses?

Review case study, Puzzling Food Poisonings in Germany. Were sufficient alternative hypotheses concerning the origin of the infection considered?

No. Attention focused immediately on Spanish cucumbers. A good critical thinker would generate a list of possible alternative causes of the outbreak, noting in which countries the bacteria infection had been found, what food products had been digested before people became ill, and where suspected food products may have been grown or processed. Such a list could be lengthy, suggesting that it would be inadvisable to focus attention on one or two lead hypotheses too early in the investigation. One approach for generating alternative hypotheses is to employ the journalists list of Who? What? Why? When? Where? and How? to stimulate ideas.

o Who: An assumption was made that the who was a farmer or group of farmers and that the outbreak was unintentional. In fact, none of the press articles challenge this assumption. However, unconsidered alternatives include terrorism or even industrial sabotage. A potential null hypothesis would be that someone caused this outbreak to occur versus it occurring naturally. When one considers the many millions of dollars in damage this outbreak caused to Spanish and other European farmers, industrial sabotage especially would have been a reasonable hypothesis to explore.

o What: At first, cucumbers were thought to have been the source. Subsequently, the list expanded to all salad vegetables and finally to bean sprouts. While alternative hypotheses were considered over the course of the outbreak, they were not considered simultaneously and tested in this way. Even the null hypothesis of not Spanish cucumbers could have been useful. Had these multiple hypothesessuch as all salad vegetablesbeen tested at the same time, attention might not have been as focused on Spanish cucumbers, and the actual source may have been found more quickly.

o When: Not applicable. o Where: As with the what question, the source of the outbreak was assumed to be

from Spain, then when that was not verified, northern Germany. A null hypothesis would have been not Spain. This could have stimulated investigators to consider other possible locations earlier. Had multiple locations been considered simultaneously, this could have also saved some time, along with the overall costs to European farmers.

o Why: Similar to the who question, motive was not considered. At the time, the outbreak could have been a deliberate attack by terrorists or an act of industrial sabotage. When it is an accident or a naturally occurring outbreak, the why question is less relevant, but it could be helpful to explore reasons for improper food processing.

o How: Consideration of the how raises several questions, such as whether people became ill because the food was infected, packaging materials were infected, or the bacteria were outside on the skin and the food was not washed.

24

Was sufficient attention paid to whether it was a version of a known strain of E. coli or an unknown strain? Yes. Scientists worked diligently to identify the particular strain of virus causing people to become ill. No premature announcements were made by German officials until the scientific work was completed.

Which techniquesituational logic, identifying historical analogies, or applying models or

theorieswould be most appropriate in generating alternative hypotheses in this situation? The lead approach in this situation would be to use historical analogies. Outbreaks like this have occurred many times in the past, and scientists have developed solid protocols for investigating the cause of the illness in the most efficient manner possible. German health officials almost certainly consulted with WHO and centers for disease control in other countries that had dealt with similar situations. Strong pressure from the public and politicians, however, can make it difficult to follow the protocols as assiduously as desired. This can lead to premature announcements that cause unnecessary harm both economically and politically.

How would an ACH exercise have helped analysts work this problem? At the outset, conducting an ACH exercise could prove problematic because the number of alternative hypotheses could be in the hundreds. One solution for this problem is to conduct an initial ACH exercise using broader categories such as 1) the strain of bacteria is human or animal based or 2) the source of contamination is an indigenous food product, an imported food product, a food processing plant, or contaminated fields where the food was grown, and 3) the point of outbreak is northern Germany, elsewhere in Germany, elsewhere in Europe, or outside of Europe. As several of these options are eliminated, the use of the ACH methodology becomes more manageable.

Chapter 14: How Do I Deal with Politicization? Review case study, Financial Crises in the United States: Chronic or Avoidable? Assume this case study was written in 2011 for generalists seeking to learn more about the US financial crisis that began in 2008. Did the author present a balanced picture of the positions advocated by the various schools of

economic thought or did the author reflect a personal political agenda? The author provided a balanced overview of the schools attempting to compare and contrast the key precepts espoused by each school. In the concluding pages of the case study, the author narrowed the discussion to four schools, but that is understandable as they appear to exert the most influence in the system. The author also provided a judgment as to which schools exerted the most influence in the crisis of the 1980s and the later mortgage crisis but supported these views with appropriate but limited evidence and argumentation. In a more exhaustive study, the author would be expected to provide more statistical data and other evidence to buttress key assertions.

25

Did the organization of the article support the thesis that the case study was a rigorous analysis and not political advocacy? The author attempted to lay out a detailed chronology of events to establish a baseline for evaluating her thesis as presented later in the paper. The author also provided an objective overview of all the various schools of thought before analyzing which were the most influential or correct in the authors opinion.

What structured analytic techniques could have the author employed to better deflect charges of politicization?

o A detailed chronology or timeline would have helped the reader capture key events and the gaps in time between them.

o A chart outlining the key assumptions attributed to each school of thought would provide some useful and possibly unanticipated insights.

o A What If? Analysis that posited, for example, that the Austrians were most correct and discussed what would happen if their views became prominent over the next ten years would help the reader understand better the most likely strategic implications of adopting this school of thought. Similarly, What If? papers could be written for the Keynesian School or the Chicago School and the comparisons across all three would be illuminating.

o Lastly, the author could have employed a Premortem Analysis and a Structured Self-Critique to ensure that there were no unfounded assumptions, logical fallacies, or major gaps in the information and the line of analysis.

If you were to include a section on opportunities analysis in the paper, what themes would you highlight? The primary theme of an opportunities analysis probably would be that the future cannot be predicted and that several future scenarios should be constructed to assess which schools of thought offer the best insights for decisionmakers. This would require developing a robust set of key drivers, a story for each scenario, and a list of indicators to track which scenarios are most likely emerging. The identification of key drivers would be particularly beneficial to policymakers signaling where it would be most effective to target new policies.

Chapter 15: How Might I Be Spectacularly Wrong? Review case study, Countering the Iranian Nuclear Threat: Stuxnet and its Broader Implications. First conduct a Premortem Assessment on the case study.

o Consider each of the following questions as it relates to the case study and write down your response to each in a sentence or two: Who? What? When? Where? Why? and How?

o Conduct a brainstorming session asking how each of the answers you generated for each question might be spectacularly wrong.

26

Who? The United States and Israel are cited as the most likely perpetrators of the Stuxnet attack, but the possibility of another nation with a sophisticated cyber capabilities or even a highly skilled hacker cannot be ruled out until more is known about the attack. Russia, China, and other Western powers such as the UK, France, and Germany probably have the cyber capability to launch the attack. Russia and China would have fewer motives, but the European countries would be at least as motivated as the United States given their greater geographic proximity. What? Research undertaken in the last two years has removed most of the initial uncertainty; we now can say with a high level of confidence that the two digital bombs were released in a direct attack on Iranian nuclear facilities and we know most of the defining elements of the attack. Could there be other code that has not been examined carefully that could indicate additional attacks were attempted? When? The attack commenced in July 2009 and continued to do damage through mid-2010. Are we certain that the Stuxnet worm is not still causing damage at locations we have not yet identified? The absence of evidence is not necessarily evidence of absence. Why? The motivation for the attack almost certainly was to disrupt the Iranian nuclear program, focusing in particular on the processes for enriching uranium sufficiently to use in nuclear weapons. Was the motivation to disrupt operations or to disable totally both facilities? If it was the latter, then the operation may be viewed by the perpetrator as relatively unsuccessful. Attention should focus on whether a more powerful attack is being planned that could have broader repercussions both within Iran and in other countries. Where? The two most obvious points of attack were the Bashehr nuclear reactor and the Natanz uranium enrichment facility. Other sites may also have been targeted, but no reports of problems arising at other facilities are noted. Our assumption that the attack was unleashed against only two targets may deserve more scrutiny. We should check our assessment of the code language to determine if other facilities in Iran and possibly in other countries might also have been targeted. How? Subsequent research by computer experts has revealed that a virus was used to change a small piece of code in a component produced by the German company Siemens that the Iranians use to manage the operation of turbines at the Bushehr reactor and nuclear centrifuges at the Natanz facility. Are we certain all the damage was done by the Stuxnet code? Could many or most of the centrifuges have been destroyed by other factors including inferior imported parts and direct sabotage? Could the Stuxnet attack been intended to mask the activities of agent saboteurs operating at both facilities?

27

Decide what additional work needs to be done to buttress the analysis in the case study and whether any key judgments need to be adjusted. The Premortem Assessment could spur additional research or collection aimed at answering the following questions:

o What nations or groups would have sufficient motive and capability to launch the Stuxnet attack?

o Can the code be examined to determine if there were any other intended targets of the attack?

o Can we confirm that Stuxnet would have caused all the damage known to have occurred?

o Is there any reporting that problems are continuing to occur at either facility or at any other facility in Iran?

Conduct a Structured Self-Critique of the analysis in the case study focusing in particular on the following questions:

o Have alternative explanations or hypotheses been considered? Have we overlooked the possibility that other factors (inferior imported equipment, sabotage) might have caused some or all of the reported damage? What if the Stuxnet attack occurred simultaneously with other attacks?

o Was a broad range of diverse opinions solicited? Most of the attention has focused on examining computer code and the vulnerabilities of controllers for industrial machinery. It might be useful to enlist some Red Team players who specialize in finding systems vulnerabilities by physically penetrating such facilities. Including regional political specialists might also provide more insight into who conducted the attack and what their ultimate objectives were.

o Were key assumptions identified and critically examined? Were unsupported assumptions identified? Did the group consider how this could change the analysis? Are we correct in assuming that only two sites were targeted by the Stuxnet worm? That the Stuxnet attack could only be launched by a country with a highly sophisticated offensive cyber capability? That the attack will significantly retard nuclear programs in Iran? That the bug at Cheech Air Force Base is more a nuisance than a threat? That the attack will not have unintended and undesirable consequences? That Iran cannot or will not retaliate in kind or some other way?

o Are there significant information gaps? Have we considered how the lack of this data could have influenced the key analytic judgments? A key information gap is the identity of the perpetrator. If the person or country that designed the worm was fairly unsophisticated, then the risk of other systems becoming infected outside of Iran could be much greater, substantially increasing the potential damage that has, or could soon be done, by the attack. Another unknown is the extent to which, if at all, the United States and Israel might have collaborated on developing the attack. If evidence later emerged regarding close and continuing collaboration, what would that imply about future actions each

28

state might take against Iran? And what would it imply if no such collaboration existed or had since ceased?

o Does the government, company, or competitor have a motive, opportunity, or means to engage in deception to influence policy or change our behavior? Many of the key players in this case study have strong motives to deceive. The state or group that launched the attack has every reason to deceive the world that it was the perpetrator of the attack. It does not want to be held responsible for the damage done, it does not want to give the Iranian government a justification to retaliate, and it could be accused by the international community of engaging in an unjustified act of war. If neither of the prime suspects were the perpetrator, then the actual sponsor has even more reason to be deceptive to avoid international opprobrium and potential retaliation. The Iranians, for their part, have reason to obscure and minimize the true impact of damage done to dissuade future attacks.

On the other hand, the computer experts quoted in the case study have no motives for not telling the truth. In fact, the more accurate their analysis, the more their reputations will be enhanced. Siemens also is highly motivated to ensure that as much is known about the attacks in order to reduce the vulnerability of its future systems to such attacks.

Chapter 16. Is My Argument Persuasive? Review case study, Financial Crises in the United States: Chronic or Avoidable? Assume this case study was written in 2011 for generalists seeking to learn more about the US financial crisis that began in 2008. What was the primary thesis or argument of the paper?

The primary thesis of this paper is that political schools of thought are important, but not the only factors that influence economic policy decision making in the wake of a financial crisis. A secondary thesis is that the current political and economic decision making process in the United States is not capable of initiating systematic structural reforms. Another secondary thesis is that exogenous political variables have undercut good economic decision making and are likely to continue to do so in the future.

How does the author establish credibility in making this argument?

The author establishes credibility by first providing a detailed chronology of past financial crises that sets a framework for analyzing how the crisis was handled. The author also establishes credibility by discussing all the key schools of economic thought before focusing on those deemed most influential.

29

Which schools does the author suggest provide the most useful insights in explaining what caused the financial crisis of 2008? Which of the five techniques does the author rely on the most to make her or his argument? The narrative and the graphics argue that the Keynesians and the Monetarists were the most influential schools in the 1980s crisis, and the Austrians and Public Choice schools gained significant influence in the more recent financial crisis. The author relies mostly on contrast to structure the paper, showing how each school of thought differs from the other and how each school of thought would recommend a substantially different policy approach. Knowing that this subject is not an easy one for most general readers to grasp, the author also relies heavily on simplicity to present the basics of each school of thought without bogging down the reader in too much economic theory or legislative detail.

To which of the seven challenges discussed in the chapter is the author most susceptible?

o The largest challenge in drafting the paper was to avoid the trap of oversimplification. Each school of thought has written multiple theses on the issues covered in this case study. Care has to be taken to generalize without sacrificing the sophistication of the baseline arguments. Given the scope and page limitations of the case study, incorporating all major aspects of each school of thought, each factor, and each influential person or group, would have been a difficult task. In addition, much of the case study focuses on the financial crisis of the 1980s to establish a baseline for analysis. This left little room to provide an in-depth discussion of the more recent dynamics beginning in 2007.

o A parallel concern is whether the broad scope of analysis made the author subject to

the trap of vagueness. Perhaps it would have been useful to ask an economist well-schooled in this topic to edit the paper sharpening the presentation of facts as well as lines of analysis. Another useful exercise would be to narrow the scope of the analysis to assess if further detail makes a meaningful contribution.

o Considering the timeliness of the case when it was written in 2011, fast moving events

in the political world are likely to make the paper out-of-date as the debate progresses, more information becomes available, and trends become clearer. The possibility of this becoming a problem is a question that can only be answered by someone now reading this case study.

o In addition, primarily due to time constraints, the challenge of familiarity was

something that could have biased the authors analysis. The analyses and schools of thought were those with which the author was already familiar. While the author is generally well versed in the spectrum of schools of thought, a more systematic literature review would have helped to reduce this bias.

o A similar challenge the author faced was satisficing. This was also due to time

constraints and is likely to be an ever-present problem in both analysis and original research. As with the challenge of familiarity, a more systematic literature review would have helped mitigate this potential problem.

30

o A good technique for reducing the potential for mismatched data or improper interpretations is to seek peer review from an outside economist (or economists). Seek out experts with differing opinions.

Chapter 17: How Should I Portray Probability and Levels of Confidence?

Review case study, Yemen: the Next Afghanistan? Draft a final paragraph or two presenting an analysis that answers the question: Will Yemen become another Afghanistan in the next five years? How would you assess the chances that Yemen will become a less reliable partner in the

counterterrorism struggle over the next five years? What words, percentages, or bettors odds would you assign to this judgment? Based on the evidence presented in the case study, a strong case can be made that the situation will get worse and not better. This prognosis could change, however, based on developments on the ground after the case study was published in 2012. That is why we recommend that the students draft a concluding paragraph or two that captures the most current developments in Yemen. Assuming that the fundamentals do not change dramatically, most students will draft a sentence stating that the chances of Yemen becoming a less reliable partner are likely, probable, between 50 and 80 percent, or two chances out of three.

What is the rationale behind your choice of a word, percentage, or bettors odd in the answer you provided above? In other words, if you added the word because to the end of the sentence above, what reasons would you give to complete the sentence? Factors that could be cited to support the assessment that Yemen will become a less reliable partner are the political pressures within the political elite, the political and social fallout resulting from the loss of oil revenue, the relative weakness of civil society, the power of the tribal elites, the lack of internal security, and the uncertainty of foreign assistance.

What level of confidence do you have in the sources used to support the analysis on whether

Yemen is likely to become another Afghanistan? Most analysts would describe their level of confidence in the sources as between medium and high. The analysis in the article is well-sourced and the author provides multiple knowledgeable sources to buttress the analytic judgments. The inclusion of reliable classified sources with good access could easily raise the level of confidence to high. If the situation deteriorates significantly and reporters and country experts are denied access to the country or to key actors or must increasingly depend on new and untested sources, then the level of confidence could drop to low.

What level of confidence do you have in your overall assessment regarding Yemens future

reliability as a partner in the counterterrorism effort? Depending on how developments unfold, the level of confidence in the analytic judgment should be medium and possibly low if current conditions deteriorate sharply.

31

Chapter 18. How Can Graphics Support My Analysis? Review case study, The End of the Era of Aircraft Carriers. Did the analyst incorporate an appropriate number of graphics in the article?

Yes, the graphics served three functions: presenting a large amount of data in the most efficient way possible, demonstrating visually the threat posed by anti-ship missiles, and summarizing the paper in a single graphic. The case study could be criticized for not including a map showing country locations, the Strait of Malacca, and the Gulf of Aden. A longer study could have included additional graphics summarizing anti-ship missile capabilities and providing national budget projections for all nations that now have aircraft carriers.

Did any graphic summarize a large amount of information in a single chart? Figure III.1 attempts to summarize three sets of data in one overarching graphic: total military expenditures for the states with the largest naval forces and aircraft carriers, military spending as a percent of Gross Domestic Product (GDP), and countries with aircraft carriers. Both vertical and horizontal bars as well as icons of aircraft carriers were used to simplify the data presentation. Since the United States is the dominant naval power, all the statistics relating to the United States are arrayed in a single column to the left to show that the United States has a larger budget and more carriers than the other twelve countries combined.

Did any graphic add meaning or provide unique insights? All three graphics met this test to some degree.

o The chart Largest Military Budgets and Active Aircraft Carriers graphically illustrates the dominance of the United States by showing it has a larger budget and more carriers than the other twelve countries combined.

o The chart of anti-ship missile ranges shows how vulnerable aircraft carrier battle groups are to new anti-ship weapons technology. It also underscores the point that anti-ship missiles can be launched from both land and sea.

o The chart Key Drivers Reducing the Size of the Global Aircraft Carrier Fleet captures in one graphic the primary thesis of the paperthree key drivers are likely to force a major reduction in the global aircraft carrier fleetand uses icons to show how the number of aircraft carriers may be reduced by as much as half in 2022.