critical thinking as scientiﬁc reasoning

C:/ITOOLS/WMS/CUP-NEW/18876406/WORKINGFOLDER/HALPERN/9781108497152C12.3D 280 [280–308] 10.8.20197:15PM

chapter 1 2

Critical Thinking as Scientific ReasoningExamining the Power of Sports Momentum

John Ruscio and Kevin BradyThe College of New Jersey

Introduction

At the start of the 2016 season in the National Football League (NFL), thePhiladelphia Eagles won their first three games. Under the leadership ofnew head coach Doug Pederson and with the promising talent of rookiequarterback Carson Wentz, they seemed unstoppable. There was endlessdiscussion in the sports media about how far the Eagles’momentummightcarry them. What nobody saw coming is that they would lose 9 of theirremaining 13 games, ending the season with a 7–9 record and missing theplayoffs. Just as the Eagles’ decline was beginning, enthusiasm shifted tothe Minnesota Vikings. They opened their season even stronger, winningtheir first five games. This, it seemed, was the team that had all themomentum and would take the league by storm. The Vikings proceededto lose eight of their remaining eleven games, ending the season with an8–8 record andmissing the playoffs. Something like this happens every yearin the NFL, and it is by no means unique to that league, that sport, or anysport. A string of victories creates the perception of momentum and anexpectation of continued success. When win streaks come crashing downto Earth, does this diminish belief in the power of sports momentum? Nota chance!Back up a couple of years, when the New England Patriots opened the

2014 season with a 2–2 record. With the legendary combination of coachBill Belichick and quarterback Tom Brady, the Patriots had been on anunprecedented run of success for more than a decade and were perennialcontenders for a Super Bowl appearance. After an embarrassing loss to theKansas City Chiefs in week 4, the sports world could talk of little else but“the end of the dynasty”. Their momentum had finally run its course, andthere were calls for replacing Belichick and benching Brady. At the sametime, the Seattle Seahawks, coming off a resounding Super Bowl victory

280


over the high-octane Denver Broncos, started their season with a 3–3record. They, too, appeared to have lost all their momentum. Despitetheir falls from exalted status, both teams would end the season withrecords of 12–4, earning the top playoff seeds in their respective confer-ences. They battled their way through tough playoff games to earn spots inthe Super Bowl, which was another hard-fought contest that came down tothe final seconds of the game, as the Seahawks opted not to give their starrunning back a chance to win the game from one yard out and MalcomButler sealed the victory for the Patriots with a goal-line interception. Dooutstanding finishes to seasons that begin with bitter disappointmentdiminish belief in the power of sports momentum? Again, not a chance.How is it that a belief can survive in the face of apparently unsupportive

evidence? Where do beliefs of questionable merit come from in the firstplace, and how do they gain traction? Rather than gullibly believing everynew idea that comes along or close-mindedly rejecting them all, what canwe do to maintain an open-minded skepticism, to think critically aboutclaims? The literature on judgment and decision making (Kahneman,2011) documents the many shortcomings in our thinking that make ussusceptible to erroneous beliefs, and the tools of scientific reasoningprovide us with means of protection (Sagan, 1995). The case of sportsmomentum will be used to illustrate our cognitive vulnerability as well asthe role of critical thinking in arriving at more accurate conclusions.

What is Sports Momentum?

Michael Kent (2006) defines sports momentum as:

The positive or negative change in cognition, affect, physiology, and beha-vior caused by an event or series of events that affects either the perceptionsof the competitors or, perhaps, the quality of performance and the outcomeof the competition. Positive momentum is associated with periods ofcompetition, such as a winning streak, in which everything seems to “goright” for the competitors. In contrast, negative momentum is associatedwith periods, such as a losing streak, when everything seems to “go wrong”(Kent 2006, p. 444).

There are two important parts to this definition. First, sports momentum issaid to consist of changes in thought, feeling, or behavior brought about bypast performance. For example, an athlete might become more confidentin his or her ability to perform well as a result of recent success, or lessconfident as a result of recent failure. Even if not at a professional level, andeven if not in the realm of sports, we have all experienced thoughts and

Critical Thinking as Scientific Reasoning 281


feelings associated with successful (as well as unsuccessful) performance ata task. There is no denying the existence of such psychological experiences.This component of sports momentum, the psychological changes causedby past performance, is not in doubt.The second part of the definition, on the other hand, is where greater

skepticism may be warranted. It is said that the psychological changesexperienced in the wake of success or failure influence either how athletesare perceived or how they perform. This is a crucial distinction. It acknowl-edges that belief in the power of sports momentum to affect performancemay be a cognitive illusion, a perception that is not matched by reality. Issports momentum merely descriptive, identifying a run of success (orfailure), or does it hold predictive value, signaling an increased likelihoodof continued success (or failure)?Most people who talk about sports momentum appear to ignore the

possibility that it is a cognitive illusion and believe instead that it holdspredictive value – that there is a causal mechanism by which past perfor-mance affects future performance. For example, an NFL team that has wonseveral games in a row is thought to bring something to their next gamethat makes them a tougher opponent than a team with a comparable win–loss record but not a run of recent success. Why does it seem that we areprimed to believe in the power of sports momentum? Why does thepossibility that it might be a cognitive illusion seldom receive seriousconsideration? Why does belief in the power of sports momentum persistdespite abundant evidence that challenges it? The answers to these ques-tions, along with others regarding a wide range of questionable beliefs, canbe found through an exploration of the cognitive heuristics, or mentalshortcuts, that predominate when we are engaged in uncritical thinking,which is our default state.

Uncritical Thinking

Kahneman (2011) describes a metaphor for human thinking that encom-passes two different systems, which for convenience he labels as System 1and System 2. Whereas System 2 is effortful, slow, and requires deliberateattention and control, System 1 is comparatively effortless, fast, and auto-matic, operating mostly outside conscious awareness. Just as it would beimpossibly taxing on the brain’s limited resources to pay conscious atten-tion to the processing of all sensory input that it receives, it would beimpossibly taxing to engage System 2 to reach every judgment or decisionin our lives, to evaluate every claim to knowledge that we encounter. Out

282 john ruscio and kevin brady


of practical necessity, most of our thinking takes place within the realm ofSystem 1.The mental shortcuts that operate in System 1, which have come to be

known as cognitive heuristics (Tversky & Kahneman, 1974), generallyserve us well, especially in terms of their efficiency. They allow us toreach judgments or decisions quickly and easily, and particularly whenwe consider information well within the confines of our everyday experi-ence. System 1 usually steers us in the right direction. There is, however,a “satisficing” quality to much of what takes place in System 1 (Simon,1956), an implicit criterion of “good enough” output that sacrifices someaccuracy in order to maintain cognitive ease. System 2 contains toolscapable of greater accuracy, but because it is not feasible to employ thesefor most of our thinking, we rely heavily on System 1. The tools of criticalthinking reside almost exclusively in System 2. Most of these need to belearned through study, honed through practice, and applied with consid-erable effort when that seems worthwhile. Because the shortcuts in System1 operate in an effortless, fast, and automatic manner, this will be ourdefault mode of uncritical thinking.If we must use mental shortcuts operating outside conscious awareness

to avoid mental exhaustion, how do we come to understand these heur-istics? Another analogy with perception holds the answer. Just as percep-tual illusions can teach us a lot about the ways that our perceptual systemswork, cognitive illusions likewise teach us a lot about the heuristics in ourSystem 1 repertoire. In a seminal paper synthesizing much of their earliestwork together, Tversky and Kahneman (1974) described three heuristicsthey discovered by observing the systematic and predictable mistakes theyand their research subjects made in a variety of tasks.

The Representativeness Heuristic

One mental shortcut of System 1 is the representativeness heuristic withwhich we estimate probability or likelihood according to perceived similarity(Kahneman&Tversky, 1972). For example, try rank-ordering the likelihoodof observing the following three ordered sequences of girl (G) and boy (B)births: (a) GGGGBBBB, (b) GBGBGBGB, (c) GBBGGGBB. Sequence (a)has an obvious pattern that makes it seem unlikely to occur by chance, andto a lesser extent so does sequence (b). Sequence (c), by comparison, is moresimilar to – or more representative of – what we expect by chance.Comparing the similarity of an observation to an expectation is not funda-mentally unreasonable, and it certainly is an efficient way to reach



a judgment. This is especially true when a more objective approach is eitherunavailable (e.g., we don’t know how to calculate probabilities) or notfeasible (e.g., we don’t care to spend the time to do so).When engaged in System 1 thinking, we rely on the representativeness

heuristic to avoid the harder work of thinking in terms of objective prob-abilities, substituting instead an easier task that can be handled quickly.Rather than calculating the probability of each G-B sequence, we form animpression of which sequences are more similar to what we expect random-ness to look like. Patterns of any kind do not appear random, and thereforethey seem unlikely to occur by chance. This cries out for an explanation. Ofcourse, all randomly generated G-B sequences of the same length are equallylikely to occur (given the assumption that 50 percent of all births aregirls and 50 percent are boys), even those as seemingly non-random asGGGGGGGG and BBBBBBBB.Belief in the power of sports momentum probably benefits from the

representativeness heuristic. A winning (or losing) streak is inconsistentwith what we expect to observe by chance. Of course, it’s well known topsychological scientists that people are biased in their attempts to produceor recognize truly random sequences (Nickerson, 2002). We intuitivelyexpect to see fewer clusters, and more alternations, than actually occur bychance. Philadelphia has won three games in a row; the Vikings have wonfive in a row: clearly these teams have the momentum on their side. That’sas far as the representativeness heuristic is likely to take us.

The Availability Heuristic

A second shortcut of System 1 is the availability heuristic with which weestimate frequency or probability according to the ease with which instancescan be recalled (Tversky & Kahneman, 1973). For example, do you thinkthat more people die from homicide or suicide in the United Stateseach year? Because few of us are familiar with objective data on the frequencyof various causes of death, and unlikely to spend time looking up the factsunless there is a compelling reason to do so, we rely on the availabilityheuristic. This substitutes a question that can be answered fairly easily:Which comes to mind more readily, instances of homicide or suicide?Whichever we can recall more easily will be judged more common.The availability heuristic tends to work well, as things that are more

frequent in our environment usually do leave behind more instances in ourmemory and are therefore easier to recall. Biases creep in when some eventsare over- or underrepresented in our memories – due to their vividness,



salience, danger, or media coverage, for example – and thereby becomeartificially easier to recall. Homicides generally attract much more mediacoverage than suicides, bringing them to the attention of a larger audience.For a variety of reasons, suicides tend to remain comparatively private.Whereas many people believe homicide to be a more common cause ofdeath than suicide, comprehensive data show the reverse. Annual reportsfrom the Centers for Disease Control show that in a typical year in theUnited States, there are more than twice as many deaths by suicide than byhomicide.Belief in the power of sports momentum probably benefits from the

availability heuristic. A winning (or losing) streak is much more salient andvivid than a mixed series of wins and losses. These streaks receive consider-able attention in sports media. This enhances our likelihood of beingexposed to such streaks, often repeatedly, and thereby of encoding themfor later retrieval from memory. When we do find ourselves estimatingtheir frequency, the availability heuristic is likely to yield an overestimatedue to our artificial overexposure to streaks. The perceived frequency ofstreaks supports a belief in the power of sports momentum.

The Anchoring and Adjustment Heuristic

A third shortcut of System 1 is the anchoring and adjustment heuristic withwhich we estimate an unknown quantity by starting with an initial valueand moving in the correct direction, but not far enough (Tversky &Kahneman, 1974). For example, take five seconds to estimate the followingproduct:

1 × 2 × 3 × 4 × 5 × 6 × 7 × 8

It is highly unlikely that you could do this arithmetic so quickly.Multiplication is a perfect example of a deliberate, effortful, and slowprocess that takes place in System 2 thinking. We must work hard tolearn the skill, and we must commit effort to use it. Even with more time,most people would find this to be a very challenging problem to solve intheir head.By using System 1 thinking, the anchoring heuristic enables us to simplify

the problem to meet the constraints of time and cognitive abilities. Youwould probably begin with some multiplication. Suppose that five seconds isenough for you to recognize that 1 × 2 is 2, 2 × 3 is 6, and 6 × 4 is 24. As timeexpires, what you have is a starting point, or anchor. You know that thecorrect answer to the problem is higher still, because there are more numbers



left to multiply, so you adjust upwards. In Tversky and Kahneman’s (1974)original study, university students’median estimate was 512 – but the correctanswer is 40,320. A partial calculation provides an anchor, but the adjust-ment does not go nearly far enough in the right direction.Further evidence that a partial calculation serves as an anchor comes from

a variant of the problem. Suppose you had been given five seconds to estimatethis product, instead:

8 × 7 × 6 × 5 × 4 × 3 × 2 × 1

If you begin with some multiplication, five seconds might be enough foryou to figure out that 8 × 7 is 56, and 56 × 6 is, well, something greater than300. Look what has happened to the anchor. Rather than 24, now it isalready over 300. When given this version of the problem, students inTversky and Kahneman’s (1974) study gave a median answer of 2,250. It isstill much too small, but it is more than four times larger than before. Thearbitrary choice of which order to write the numbers in the problemchanged the anchor, which in turn affected the final estimates. Researchhas demonstrated that anchors can be obtained in many ways, includingtransparently arbitrary methods such as random spins of a wheel orreference to digits in a personal ID number or less obviously biasedmethods such as partial calculations or retrieval of memories (Furnham& Boo, 2011). Whatever the source of an anchor, the adjustment tends tobe insufficient in magnitude.Belief in the power of sports momentum is likely to benefit from the

anchoring and adjustment heuristic. The anchor in this instance is recentexperience, either personal or vicarious, involving a winning (or losing)streak. An estimate of future performance is not adjusted sufficiently farfrom this anchor of extremely good (or poor) performance. Even though werecognize it is unlikely the Eagles or Vikings will remain undefeated througha full sixteen-game season, we fail to adjust far enough away from their earlyrun of success and end up surprised by the more ordinary record that follows.This also constitutes a failure to understand regression to the mean, anotherconcept that is entirely foreign to the cognitive heuristics of System 1.

Other Characteristics of System 1

The representativeness, availability, and anchoring and adjustment heur-istics are not the only ways that System 1 helps us cope with complexityand time constraints. Kahneman (2011) describes many additional char-acteristics of System 1, such as distinguishing the surprising from the



normal, inferring and inventing causes and intentions, neglecting ambi-guity and suppressing doubt, and being biased to believe and confirm.Once again, the characteristics of System 1 are apt descriptions of uncri-tical thinking because they promote a relatively blind acceptance ofbeliefs based on minimal evidence. At the same time, however, they areinvaluable tools.The ability to identify patterns and assign meaning to them almost

certainly holds great survival value, as this can alert us to potentialthreats or opportunities and impel us to act. That some of thesepatterns will be cognitive illusions that prompt us to take unnecessaryaction demonstrates only that we may be biased to err on the side ofcaution. In dealing with some false-positive beliefs that turn out notto be threats or opportunities, little may be lost in the process. On theother hand, the absence of highly sensitive pattern-recognition andmeaning-making abilities could be more costly, causing us to miss outon opportunities or to take defensive action in the face of real threats.It is far too cumbersome to engage in critical thinking continuously;this would literally deplete scarce mental resources (Kahneman, 2011).A default mode that excels at identifying patterns and ascribingsignificance to them, and that does this with great ease, is an excep-tional achievement in cognitive architecture.It turns out that a great many characteristics of System 1 help to

understand how dubious ideas, such as belief in the power of sportsmomentum, can emerge and spread. A winning or losing streak issurprising, relative to more normal fluctuations in performance. Itstands out as a pattern that needs to be explained, and the notion ofsports momentum fills this void perfectly. By biasing our thinkingtoward belief (e.g., that past performance exerts an influence on futureperformance), seeking confirmation (e.g., noticing other streaks,remembering instances where streaks persisted) while suppressingambiguity and doubt (e.g., forgetting instances where streaks ended,not worrying too much about how it is that the past performanceplays a causal role in future performance, nor about the possibilitythat streaks themselves might occur by chance), confidence in thepower of sports momentum can increase. Between the three heuristicsand many additional characteristics of System 1, our uncritical think-ing primes us to perceive clustered series of outcomes rather thanrandom sequences of wins and losses, to believe that momentum istheir causal impetus, and to become confident in this belief ratherthan submitting it to careful scrutiny or rigorous tests.



Critical Thinking

With so many forces aligning to create and bolster support for a dubiousbelief such as the power of sports momentum, it’s clearly also of value tobe capable of critical thinking. If System 1 operates primarily as uncriticalthinking, we must turn to System 2 as a source of critical thinking.However, simply exercising voluntary control and devoting attentionand effort to our thought process does not necessarily enable us toreach more accurate conclusions via System 2 than we would with aneasy reliance on System 1. If you are unable to multiply large numbersin your head, for example, you can devote all the time you like to System2 thinking but you will still fail to calculate correctly that 1 × 2 × 3 × . . . × 8= 40,320.To assist in critical thinking, System 2must be equipped with tools that

effectively combat our mental shortcomings, such as our cognitive limita-tions and biases. This is just what scientific methods have been designed todo, and to the extent that we can successfully learn to engage in morescientific reasoning, there is hope that we can overcome the tendencies tooveridentify patterns, infuse them with too much meaning, and hold themwith undo confidence. These tools do not come naturally to most people.Anyone who has studied, let alone taught, even the most fundamentalprinciples of research design, measurement, or statistics knows how chal-lenging it can be to master these skills. These are among the tools that mustbe added to the repertoire of System 2 so that they can be used to gatherpertinent information and examine it systematically in order to test ideasrigorously. Thus, we define critical thinking as scientific reasoning, asequipping System 2 to use an array of human inventions specificallydesigned to overcome the inherent limitations and biases that lead toefficient but error-prone System 1 thinking.

Is It Worth the Effort?

Thinking outside the lab as you have learned to think in the lab canimprove your judgments and decisions, and can help to evaluate themerit of claims to knowledge. Just as using System 2 for any other purposeis highly effortful, the same will be true of critical thinking. Is it worth thecommitment of resources? Will it be worth the trouble to think hard aboutthe logic and evidence bearing on a questionable belief? What if thisrequires gathering more comprehensive information and examining it foryourself? If critical thinking requires that new tools be added to the



repertoire available to System 2, will it be worth the time and effort to learnthem, to practice them, and to apply them carefully? Not everyone will bewilling to do so.In addition to cognitive skills, there are also tools that can extend the

observational powers of our senses (e.g., microscopes, telescopes, infraredor ultraviolet cameras, time lapse or slow-motion video playback), theretrieval of information beyond what is accessible in our memory (e.g.,archival records), or the computational power of our brains (e.g., calcula-tors, spreadsheets for data management, software for statistical analysis).The proper use of these tools must be learned. The investment in learningto work with such hardware and software can be enormously helpful togather and examine a wider range of relevant information, and takingadvantage of the speed and power of computers allows us to perform manytasks more efficiently and accurately. These tools are not substitutes forcritical thinking, they are essential components.Whereas uncritical thinking relies on the effortless, fast, and automatic

processes of System 1, critical thinking requires the will to commit time andeffort to the more demanding processes of System 2. Moreover, it alsorequires that we have equipped System 2 with the necessary tools. Some ofthese tools, in turn, will require external assistance. An informative statis-tical analysis, for example, is unlikely to be possible without at least usinga calculator plus a reference guide to supply the appropriate formulas andtables, if not computer software to enable more complex analyses of largerdata sets. Often, then, there will be layers of special-purpose tools thatdemand effort-intensive steps to learn and to apply on a case-by-case basis.To justify all of this effort, there must be something significant at stake.

Those who aspire to a career in a scientific discipline devote enormousamounts of time and energy to learning the tools of the trade. The stakesare professional: obtaining a job, securing funding, publishing research,earning promotions, attaining honors and awards, and so forth.Nonscientists, or scientists operating outside their areas of expertise, mayhave considerably less incentive to acquire the skills and master the toolsrequired to think critically about a particular claim to knowledge. Whenthere is relatively little at stake, when there are few or no personal con-sequences attached to the accepting a belief, rejecting it, or remainingundecided, it should not be surprising that questionable beliefs will faceonly the uncritical thinking of System 1.It is possible that belief in the power of sports momentum persists in part

for this very reason – that neither holding nor rejecting this belief is verycostly. Other questionable beliefs in the realm of sports have been changed



in response to critical thinking when the stakes became high. For example,as Lewis (2003) described so vividly in Moneyball, there were significantflaws in the conventional methods used to assign value to baseball players.Scouts and coaches looked for certain physical attributes and prized certainstatistical accomplishments, but it turns out that more objective analysesusing improved data sources provided better measures of value. Thisenabled savvy teams to identify undervalued players. Lewis chronicles theremarkable success of the Oakland Athletics, who leveraged one of thelowest payrolls in all of Major League Baseball (MLB) to field teams thatcompeted successfully with those of the highest payrolls. The driving forcebehind this revolution in baseball thinking was Billy Beane. When hedeclined an offer to become the General Manager (GM) of the Boston RedSox in 2002, they turned instead to Theo Epstein, who became the young-est GM in MLB history. Epstein ran with Beane’s ideas and achieved evengreater success. In 2004, the Red Sox won their first World Series in 86years, and they have won it three more times since then. In 2011, Epsteinbecame President of the Chicago Cubs, and in 2016 they won their firstWorld Series in 108 years.Throughout the world of professional sports, it is becoming clear that

bringing in talented data analysts can help to build a stronger roster as wellas make better-informed coaching decisions. This revolution in decisionmaking has happened precisely because there is so much at stake inprofessional sports. The same will hold true for certain college sports,too, where there is a lot of money on the line (e.g., athletes’ prospects forprofessional careers, college coaching salaries, college revenue streams suchas television revenues or fundraising). It won’t be surprising to learn thatdata-driven strategies assist in the recruitment of talented athletes or inmaking game-time decisions. It would be surprising to see this approachtaken in a local youth sports league, as the potential rewards simply do notjustify the effort.Something similar – the lack of significant consequences – may explain

why beliefs about sports momentum are accepted with little or no scrutiny,why they’re rarely submitted to rigorous tests, and why they are so resilientto challenge. The psychological experience of changes in feelings, thoughts,and behaviors following success or failure is real, andmost people can relatepersonally to this aspect of momentum even if not as professional athletes.The question of whether these changes affect subsequent performance maynot actually matter very much. Winning or losing streaks seem special, andall the forces of uncritical thinking push in the same direction, to accept thepower of sports momentum.



Even if all the talk about sports momentum is grounded in nothingmore than a cognitive illusion, perhaps the only people eager to knowthe truth would be those with something significant at stake. Forexample, anyone wagering on sports might take an interest in thissubject. Betting lines are responsive to the beliefs of those placingbets, whether they are well-informed or otherwise. If most gamblersbelieve in the power of sports momentum and it turns out this isa cognitive illusion, this could create a profitable betting opportunity.The general strategy would be to bet against streaks because the bettinglines may be skewed by a misconception. For example, after winningtheir first three games in 2016, the Eagles were favored by 3 points intheir fourth game; they lost by 1. Likewise, after winning their first fivegames, the Vikings were favored by 3 points in their sixth game; theylost by 11. Betting against both teams at these times would have paidoff. Of course, these examples were not selected at random and maynot be representative. Bettor beware! We are simply speculating that ifbelief in sports momentum is unwarranted but still influences bettinglines, one might find that profitable opportunities emerge as teams goon winning or losing streaks (see Gandar et al., 1988 for more onrationality in NFL betting markets). By the same token, if you believein the power of sports momentum and it turns out this is mistaken,you might be putting yourself at a disadvantage when you place bets.Our point is not to encourage gambling, but to demonstrate thatcritical thinking should be heightened as stakes are raised.If scientific reasoning is to overcome the limitations and biases inherent

in System 1 thinking, what are the skills and tools that System 2 needs touse? We review five major components of scientific method that can beused to promote critical thinking, describe how they deal with the short-comings of System 1, and illustrate them through an examination of sportsmomentum. In particular, we examine the belief that winning or losingstreaks affect the likelihood of future success for NFL teams.

Consider Alternative Explanations to Establish Competing Predictions

System 1 is biased to believe, and to do this it tends to seek only con-firmatory evidence that breeds confidence. Alternative ideas may be sup-pressed, and an important countermeasure is to actively constructalternative hypotheses that make competing predictions. This means ask-ing questions that do not spring to mind by carefully considering theimplications of a belief.



Sports momentum clearly entails changes in one’s psychological experi-ence following success or failure, but it is less clear that these changes will inturn affect performance. What are the causal mechanisms? It might be thatconfidence is the key, that a rise or fall in one’s feeling of self-efficacy exertsan impact on performance. Those familiar with the Yerkes-Dodson lawmight speculate that the boost to confidence nudges arousal closer to theoptimal level, with the predicted effect being improved performance.Likewise, a fall in confidence could push arousal away from the optimallevel and lead to a corresponding drop-off in performance. Plausible as thatmight sound for behavior unfolding in the moment, it is not obvious howit might be relevant to games played on a weekly basis. Levels of arousalmight fluctuate within a game, but would this be sustained between games?Doesn’t the week-long effort of preparing for the next game dissipate thefleeting arousal effects of the earlier wins or losses? Doesn’t the immediacyof that next game, as it begins, provide its own motivation to perform andboost arousal? Can athletes even make it into the ranks of professionals ifthey require an artificial boost to approach their optimal level of arousal?Perhaps it is not arousal levels that are the causal mechanism for

a between-game momentum effect, but an effect on preparation andstrategy. Maybe winning or losing streaks prompt a different approach tothe next game than a recent history of mixed results? A team on a winningstreak might try to continue its success by doubling down on what appearsto be working. That might seem reasonable enough, but then again itmight be a self-defeating strategy. If a winning team sticks with a style ofplay that has succeeded, their opponents learn what to expect and candevelop ways to mute its impact. Every innovation in strategy is eventuallymet with an effective response. Even teams that rely heavily on the talent ofstar players learn that opponents find ways to handle them. Film study andtailoring a game plan to a specific opponent are a huge part of what it takesto win in the NFL, and teams devote a tremendous amount of energy tothese tasks. Sticking with what has worked can also be accompanied bycomplacency, a lack of discipline, and reduced dedication to workouts andpractice. None of this is likely to lead to sustained success.Likewise, it is not hard to imagine that a team on a losing streak

might try something new to break the spell of failure, and this may bemore effective. Coaches and players alike know that their careersdepend heavily on winning, and they are highly motivated to turnthings around when they begin to lose. Teams on a losing streak areunlikely to stick with what is not working for long and devising newapproaches will keep opponents guessing and make it harder for them



to game-plan effectively. Just as the behavioral responses to a winningstreak might bring it to an end sooner than expected, the behavioralresponses to a losing streak might do the same.All of this is merely speculation, but the point is that by thinking

through the implications of sports momentum one might come to doubtits potency. This requires the effort of critical thinking, not the facileacceptance of uncritical thinking. In the case of sports momentum, thecompeting predictions would be that it either (a) has the power to influ-ence later performance, in which case one would expect more and longerwinning and losing streaks to occur than would be expected by chance, or(b) does not have this power, that it is a cognitive illusion, in which caseone would expect outcomes to follow chance-level patterns.

Collect Data as Systematically and Comprehensively as Possible

System 1 relies heavily on introspection, the examination of personalexperience and the recollection of instances that appear relevant but thatare likely biased due to the search for confirmatory examples. To counterthese biases, it is imperative to gather a wider range of pertinent informa-tion and to do so as systematically as possible. It is easy to think of NFLteams that have gone on winning or losing streaks, but it is more importantto consider all teams’ performance to see whether there are in fact more andlonger streaks than expected by chance. This clearly exceeds the capacity ofSystem 1 and means turning to archival records and using tools such asspreadsheets to organize and manage these data.To examine the streakiness of NFL teams, we obtained the specific

sequences of wins and losses for every team in every year from 1978,when the NFL moved to a sixteen-game season, through 2016, the latestseason that had been completed when these data were collected. Thisyielded n ¼ 1; 169 team-seasons, which affords a fairly comprehensiveexamination of between-game momentum in the NFL.

Establish a Specific, Replicable Protocol

The uncritical thinking of System 1 is free to search for apparent patternsand supportive evidence, which makes it quite easy for questionable beliefsto emerge and to be held with confidence. Critical thinking requiresconstraints such as clear definitions of relevant constructs and a prioriplans for research design and data analysis that are expressed in sufficientdetail that they can be evaluated on their merits and replicated if desired.



The goal is to remove any wiggle room, to make it harder to ignore themisses and count the hits when tallying results to reach a conclusion.Before performing any analyses of our NFL game data, we specified the

competing predictions made by the sports-momentum and cognitive-illusion hypotheses, decided on the criteria for trimming the sample, andplanned all of the analyses that would be performed. Trimming the samplewas required because data analysis required that each team-season con-tained the same number of games, that there was variability within eachsequence, and that the outcomes were either a win or loss for each game.Therefore, we removed data for the strike-shortened seasons in 1982 and1987, for the 2007 New England Patriots (who won all sixteen games) andthe 2008 Detroit Lions (who lost all sixteen games), and for any team-season in which there were any tied games. The final sample containsn ¼ 1; 075 team-seasons. This left most (92 percent) of the original sampleintact, and because we removed data using objective criteria that havenothing to do with the pattern of wins and losses within team-seasons, thisshould not introduce any bias.As noted earlier, our competing predictions stemmed from the

hypotheses that sports momentum either does or does not have thepower to influence later performance. We planned a series of five tests,with competing predictions for what we will refer to as a momentumeffect and a cognitive illusion. The first test involves the influence of a byeweek. The NFL’s regular season spans seventeen weeks, but each teamplays only sixteen games because they will have one week off. It is widelybelieved that this bye week, usually scheduled between weeks 4 and 12,will disrupt any momentum a team may have developed, making itharder to continue a winning streak and easier to end a losing streak.The prediction for a momentum effect is that streaks will be disrupted bybye weeks, whereas the prediction for a cognitive illusion is that byeweeks do not affect streaks.The second test involves the number of runs (streaks) within team-seasons.

A team that wins half of its games could do so in a number of ways, such asWWWWWWWWLLLLLLLL orWWLLLWLWWLWWLWLL. The firstsequence contains a run of eight wins followed by a run of eight losses, fora total of two runs. The second sequence contains a run of two wins, a run ofthree losses, a run of one win, and so on, for a total of ten runs. The predictionfor a momentum effect is that the number of runs will be smaller thanexpected by chance, as many of the wins and losses will be clustered together.The prediction for a cognitive illusion is that the number of runs will matchchance-level expectations.



The third test involves autocorrelations. This examines the sequentialdependence of outcomes, in this case whether wins predict wins and lossespredict losses in back-to-back games. Figure 12.1 shows how back-to-backgames are paired to form 15 data points within a team-season. A correlationis then calculated by treating each win as a 1 and each loss as a 0. For thefirst sequence shown above, this analysis yields r ¼ :88 because winsstrongly predict wins and losses strongly predict losses. For the secondsequence, this analysis yields r ¼ �:20, suggesting little relationshipbetween wins and losses. The prediction for a momentum effect is thatautocorrelations will be larger than expected by chance, whereas the pre-diction for a cognitive illusion is that autocorrelations will match chance-level expectations.The fourth test involves predictions. If sports momentum influences

performance, this implies that teams’ records in more recent game out-comes should exert a larger influence on predictability than game outcomesin the more distant past. As a measure of predictability, we calculated theprobability that the teams with the better records won their games ata particular point in the season. Records were calculated using varyingnumbers of prior weeks’ game outcomes. The prediction for a momentumeffect is that game outcomes will be more predictable using smaller, morerecent samples of teams’ wins and losses, whereas the prediction for

Figure 12.1 Autocorrelations for two illustrative team-seasonsThe panel on the left shows a sequence of eight wins followed by eight losses. This

sequence is copied across all rows so that each pair of back-to-back games can be visualizedseparately. These pairings constitute the data points for the autocorrelation analysis, whichyields a very large positive correlation because wins tend to follow wins and losses tend tofollow losses. The panel on the right shows a sequence in which eight wins are mixed witheight losses in a more haphazard way. The correlation is small and negative, suggesting that

wins and losses are distributed fairly randomly throughout the sequence.



a cognitive illusion is that game outcomes will be more predictable usinglarger samples of teams’ wins and losses that stretch back farther, as basicstatistical theory would suggest.The fifth and final test involves the frequency of winning and losing

streaks. Each team-season can be broken into a series of runs, as shownearlier, and the frequency of such winning or losing streaks of varyinglengths across all team-seasons can be tallied. The prediction fora momentum effect is that winning and losing streaks will be longer thanexpected by chance, whereas the prediction for a cognitive illusion is thatthe lengths of streaks will be distributed according to chance-levelexpectations.

Consider the Role of Chance

System 1 is outstanding at identifying patterns and ascribing significance tothem. Many patterns identified when thinking uncritically turn out to beillusory, though. To think critically, we must consider the possibility thatmere coincidence is at work. Sometimes this can be accomplished byengaging System 2, but often it will require the use of an appropriatestatistical analysis to help assess the probability that the pattern in questionmight occur by chance.If sports momentum is nothing more than a cognitive illusion, this

means that the winning and losing streaks thought to be evidence ofmomentum effects are no more common than would be expected bychance. In principle, that sounds like a simple comparison to make, butin practice it is not so easy to establish what one might expect by chance.For example, consider what is perhaps the best-known empirical work onsports momentum, the study of the “hot hand” in basketball (Gilovich,Vallone, & Tversky, 1985). Among other things, Gilovich, Vallone, andTversky examined the probabilities that basketball players made their nextshot after havingmade 1, 2, or 3 previous shots as well as after havingmissed1, 2, or 3 previous shots. It turns out that even if the “hot hand” isa cognitive illusion, the expected probabilities are not equal across theseconditions. Miller and Sanjurjo (2016) describe a subtle but nontrivial biasin traditional measures of conditional dependence. The solution to thisproblem is to compare observed results not to a null hypothesis based onone’s intuition, but to empirical results for artificial comparison data thatare generated using a process that generates truly random sequences.In our analyses of NFL game data, we generated artificial comparison

data by holding constant the number of wins in a team-season and



randomly shuffling the order of the wins and losses. Doing this across all1,075 team-seasons creates one sample of artificial comparison data.Depending on the type of analysis being performed, we generated any-where from 10 to 1,000 such samples of artificial comparison data toobserve the results expected by chance empirically.

Weigh All Available Evidence

System 1 is biased toward seeking support for a belief, of retrievinginstances consistent with a potential pattern, and of suppressing doubt.An essential component of critical thinking is to consider all of theavailable evidence even-handedly. This involves asking whether resultsconverge in support of a conclusion, or whether they are mixed. Thisdoes not mean that all sources of information should necessarily be givenequal weight, but it does mean that even discrepant findings cannot beignored outright. As in a meta-analysis, greater weight might be given tolarger, better-controlled studies than smaller studies with poorer controls.If the support for a belief consists of anecdotes, even one who rejects thisbelief needs to account for the anecdotes. For example, one might be ableto argue persuasively that the anecdotes are exaggerated, biased and unre-presentative, misunderstandings, or perhaps even fabrications. Other cri-tical thinkers can, in turn, weigh the plausibility of your explanation forany apparently discrepant evidence.In the case of between-game momentum in the NFL, the evidence begins

with the anecdotal observation of winning and losing streaks, such as thebeginning of the 2016 season for the Eagles and Vikings, but we expand theevidence base by conducting the five tests outlined earlier. In the first test, wefound no evidence of an influence of bye weeks on streaks. Table 12.1summarizes analyses of the probability of winning after having won(or lost) varying numbers of games across two conditions: the specifiedsequences of wins (or losses) itself, or this same sequence plus a bye week.None of these analyses yielded statistically significant results, which fails tosupport a momentum effect and is consistent with a cognitive illusion.In the second test, we found no fewer runs than expected by chance.

Table 12.2 summarizes analyses including all 1,075 team-seasons as wellas subsamples based on varying numbers of wins to check the possibilitythat streakiness might emerge only with sufficiently variable outcomes(e.g., closer to 8 wins than to the extremes of 0 or 16 wins). None ofthese analyses yielded statistically significant results, which also fails tosupport a momentum effect and is consistent with a cognitive illusion.



In the third test, we found no larger autocorrelations than expectedby chance. Table 12.3 summarizes analyses including all 1,075 team-seasons and, once again, subsamples based on varying numbers ofwins. None of these analyses yielded statistically significant results,which once again fails to support a momentum effect and is consis-tent with a cognitive illusion.

Table 12.1 Does a bye week affect the probability of winning?

Sequence (X) p(win | X) n p(win | X + bye) n z p

1 win .509 3,618 .536 192 −0.73 .4652 wins .548 1,605 .560 109 −0.23 .8183 wins .550 766 .600 55 −0.73 .4684 wins .567 356 .690 29 −1.28 .2005 wins .636 162 .667 18 −0.26 .7966 wins .654 81 .600 10 0.34 .7347 wins .634 41 1.000 3 −1.29 .1978 wins .727 22 1.000 1 −61 .5441+ wins .537 7,652 .570 423 −1.32 .1882+ wins .566 3,591 .597 231 −0.92 .3573+ wins .584 1,755 .631 122 −1.02 .3074+ wins .613 865 .667 63 −0.85 .3955+ wins .638 436 .667 33 −0.34 .7386+ wins .647 224 .667 15 −0.15 .8797+ wins .632 114 .800 5 −0.77 .4438+ wins .673 55 .500 2 0.51 .6111 loss .500 3,633 .550 191 −1.33 .1842 losses .472 1,601 .427 96 0.86 .3903 losses .423 724 .509 55 −1.25 .2124 losses .336 351 .500 26 −1.69 .0915 losses .372 191 .350 290 0.19 .8486 losses .307 101 .333 6 −0.14 .8927 losses .246 57 .600 5 −1.70 .0898 losses .353 34 .500 4 −0.58 .5641+ losses .460 7,640 .493 410 −1.28 .1992+ losses .417 3,614 .443 219 −0.75 .4553+ losses .372 1,811 .455 123 −1.85 .0644+ losses .335 971 .422 64 −1.43 .1545+ losses .339 546 .368 38 −0.37 .7106+ losses .323 303 .389 18 −0.58 .5657+ losses .301 166 .417 12 −0.84 .4038+ losses .337 92 .286 7 0.28 .782

A bye week is an off week on a team’s schedule during which they do not play a game. TheNFL schedule contains seventeen weeks, but each team plays 16 games, with one bye weekthat’s usually scheduled between weeks 4 and 12.



In the fourth test, we found that predictability increased with thenumber of prior games’ outcomes included in the analysis. Figure 12.2shows that this increasing pattern is observed for each week’s games,beginning mid-season (to allow enough prior games for an informativeanalysis) and continuing through the end of the regular season and ulti-mately the Super Bowl. The sample sizes become smaller in the postseasonbecause most teams don’t make it that far, but the results nonetheless trendin the same direction. The fact that including more games, by reachingfarther back into the past, increases predictability relative to smaller sam-ples of more recent games is consistent with a cognitive illusion and notwith a momentum effect.In the fifth and final test, we found no longer or more frequent winning

or losing streaks than expected by chance. Figure 12.3 shows thatthe distributions of both types of streak closely match chance-levelexpectations. Goodness of fit tests between the observed frequencies andchance-level expected frequencies for these two distributions yieldedχ 2ð9Þ ¼ 5:13; p ¼ :177, and χ 2ð9Þ ¼ 5:36; p ¼ :198, respectively.This fails to support a momentum effect and is consistent witha cognitive illusion.

Table 12.2 Are there fewer runs than expected by chance?

Team-Seasons n Observed M Expected M (SD) z p

All 1,075 7.92 7.86 (2.14) 0.03 .9761 win 10 2.90 2.88 (0.32) 0.06 .9532 wins 32 4.47 4.50 (0.77) −0.04 .9723 wins 36 6.00 5.88 (1.13) 0.11 .9134 wins 79 6.96 7.00 (1.42) −0.03 .9765 wins 83 7.89 7.87 (1.64) 0.01 .9916 wins 97 8.60 8.51 (1.80) 0.05 .9627 wins 122 8.83 8.87 (1.90) −0.02 .9818 wins 130 8.90 9.01 (1.93) −0.06 .9569 wins 126 8.98 8.87 (1.90) 0.06 .95410 wins 120 8.88 8.50 (1.81) 0.21 .83411 wins 95 7.98 7.87 (1.65) 0.06 .94912 wins 78 7.21 7.00 (1.42) 0.15 .88413 wins 40 5.75 5.87 (1.12) −0.11 .91214 wins 21 4.38 4.50 (0.76) −0.16 .87215 wins 6 3.00 2.87 (0.33) 0.38 .704

Each mean (M) and standard deviation (SD) is for the number of runs per team-season.Expected values were calculated using 1,000 samples of artificial comparison data andtreated as population parameters for the z tests.



It’s easy to see that all five of these test results do in fact converge ona conclusion, namely, that sports momentum appears to be a cognitiveillusion. The anecdotal evidence of streaks is not difficult to account for,either, as it is entirely subsumed in the sample of data that we collected andanalyzed. The anecdotes were isolated instances of streaks that are bound tooccur. A more comprehensive examination of all teams, across all seasons,shows that there is nothing particularly streaky about NFL team perfor-mances from week to week. Winning and losing streaks occur about asoften, and persist for about as long, as you would expect by chance.Identifying such streaks serves only to describe past performance andappears to hold no predictive value for future performance. In sum, asinteresting as it might be to discuss which teams are hot or cold, there is nojustification for suggesting that it is harder to face a hot team or easier toface a cold team – at least not beyond what their cumulative win–lossrecord would otherwise suggest.Having said this, a final component of critical thinking as scientific

reasoning is to acknowledge limitations in the evidence and the extent towhich these conclusions can safely be generalized. Despite our best effortsto gather as much pertinent data as we could and to search for evidence of

Table 12.3 Are outcomes for back-to-back games positively correlated?

Team-Seasons n Observed M Expected M (SD) z p

All 1,075 −.08 −.07 (.25) −0.04 .9661 win 10 −.06 −.06 (.02) −0.07 .9412 wins 32 −.05 −.07 (.20) 0.10 .9203 wins 36 −.08 −.07 (.23) −0.05 .9634 wins 79 −.06 −.07 (.25) 0.01 .9945 wins 83 −.07 −.07 (.25) −0.03 .9806 wins 97 −.08 −.07 (.26) −0.05 .9597 wins 122 −.06 −.07 (.26) 0.02 .9838 wins 130 −.05 −.07 (.26) 0.05 .9589 wins 126 −.08 −.07 (.26) −0.05 .95910 wins 120 −.12 −.07 (.26) −0.22 .82411 wins 95 −.08 −.07 (.25) −0.06 .95312 wins 78 −.10 −.07 (.25) −0.14 .89013 wins 40 −.09 −.07 (.23) −0.09 .92514 wins 21 −.02 −.07 (.20) 0.23 .82215 wins 6 −.07 −.06 (.02) −0.38 .708

Notes. Each mean (M) and standard deviation (SD) is for the autocorrelations within eachteam-season. Expected values were calculated using 1,000 samples of artificial comparisondata and treated as population parameters for the z tests.



12

34

56

78

0.450.550.650.75

Gam

e 9

(n =

107

5)

Num

ber

of W

eeks

p(Rw > Rl)2

46

8

0.450.550.650.75

Gam

e 10

(n =

107

5)

Num

ber

of W

eeks

p(Rw > Rl)

24

68

10

0.450.550.650.75

Gam

e 11

(n =

107

5)

Num

ber

of W

eeks

p(Rw > Rl)

24

68

10

0.450.550.650.75

Gam

e 12

(n =

107

5)

Num

ber

of W

eeks

p(Rw > Rl)

24

68

1012

0.450.550.650.75

Gam

e 13

(n =

107

5)

Num

ber

of W

eeks

p(Rw > Rl)

24

68

1012

0.450.550.650.75

Gam

e 14

(n =

107

5)

Num

ber

of W

eeks

p(Rw > Rl)

24

68

1012

14

0.450.550.650.75

Gam

e 15

(n =

107

5)

Num

ber

of W

eeks

p(Rw > Rl)

24

68

1012

14

0.450.550.650.75

Gam

e 16

(n =

107

5)

Num

ber

of W

eeks

p(Rw > Rl)

510

15

0.450.550.650.75

Wild

card

(n =

251

)

Num

ber

of W

eeks

p(Rw > Rl)

510

15

0.450.550.650.75D

ivis

ion

al(n

= 2

85)

Num

ber

of W

eeks

p(Rw > Rl)

510

15

0.450.550.650.75

Ch

amp

ion

ship

(n =

144

)

Num

ber

of W

eeks

p(Rw > Rl)

510

15

0.450.550.650.75

Su

per

Bo

wl

(n =

72)

Num

ber

of W

eeks

p(Rw > Rl)

Figure12.2

Dosm

allersamplesof

morerecent

gamespredicto

utcomesbetterthan

largersamplesthatinclud

emoredistantg

ames?

The

firstgraph

show

sthe

predictabilityofGam

e9acrossallteam-seasons

usingthetwoteam

s’recordsintheprevious

1,2,3,...,8games.T

heyaxisplotsthe

probabilitythattheteam

thatwon

thegamehadabetterrecord

than

theteam

thatlostthegame.Largervaluesrepresentgreater

predictability.Solid

circlesshow

theobserved

values,accom

panied

byaverticallin

erepresentin

gthestandard

error.Chance-levelexpectatio

nsweregeneratedusing10

samplesof

artifi

cialcomparisondataandplottedasxs.


streakiness in many ways, it is possible that a momentum effect escapeddetection in our analyses. We compiled data for nearly 40 years of NFLgames, so in that regard our sample is both large and representative.However, longer sequences would provide greater statistical power. Thesixteen-game NFL schedule provides fairly short sequences within whichto search for deviations from chance-level ordering of wins and losses.A small momentum effect may have been missed.It is also possible that some observed winning or losing streaks might be

driven by true momentum effects even as many others are not, in whichcase the presence of the latter could mask the existence of the former instatistical tests. Even if this is the case, it would imply that the momentumeffect must be fairly small, and it remains to be demonstrated whetheranyone could successfully differentiate between true momentum andcognitive illusion for particular winning or losing streaks.

Summary and Conclusions

Because it would be mentally exhausting to think critically all of the time,our default mode is uncritical thinking. As Kahneman (2011) notes in hismasterpiece Thinking, Fast and Slow, the fast thinking of what he callsSystem 1 is what allows us to process information efficiently. System 1 isadept at pattern recognition and heavily biased toward belief rather thandoubt. The extensive literature on judgment and decision-making

2 4 6 8 10 12 14

050

015

00

Winning Streaks

Streak Length

Fre

quen

cy

2 4 6 8 10 12 14

050

015

00

Losing Streaks

Streak Length

Fre

quen

cy

Figure 12.3 Are long streaks observed more often than expected by chance?Solid circles show the total number of streaks of each length across all team-seasons.

Chance-level expectations were generated using 10 samples of artificial comparison data andplotted as open circles.



documents the many cognitive heuristics that generally serve us well butare prone to systematic biases. When the stakes are high enough to warrantthe added effort, engaging in critical thinking can help us attain greateraccuracy. This requires training System 2 with the tools of scientificreasoning and using them properly on a case-by-case basis.Critical thinking, undertaken through a series of five components of

scientific reasoning, leads us to suggest that sports momentum probablycarries little, if any, significance between NFL games. At the same time, wehesitate to generalize to other contexts, such as momentum effects withinNFL games or in other sports. For example, it is possible that momentummay operate within NFL games, affecting performance during the courseof one particular contest even if it does not carry over into the followingweek. Two studies have failed to find evidence of such within-gamemomentum in the NFL (Fry & Shukairy, 2012; Johnson, Stimpson, &Clark, 2012), but further research could alter that conclusion.As noted earlier, the best-known empirical work on sports momentum

began with Gilovich, Vallone, and Tversky’s (1985) study of the hot handin basketball. They concluded that this is a cognitive illusion. MoskowitzandWertheim’s (2011) chapter on momentum discusses related research inmany other sports. A meta-analysis of 27 studies and 56 effect sizes drawnfrom 9 different sports – basketball, golf, baseball, billiards, volleyball,bowling, darts, handball, and soccer, in decreasing order of numbers ofeffect sizes obtained – corroborates their conclusion (Avugos et al., 2013).Avugos et al. tested various moderator effects as well, finding no practicallyor statistically significant difference for players vs. teams, between-game vs.within-game analyses, or individual vs. team sports. More recently, Millerand Sanjurjo (2016) challenged the null results for the hot hand in basket-ball on the grounds that Gilovich, Vallone, and Tversky’s measures ofstreak shooting weren’t sufficiently sensitive and at least some of theirstatistical tests contained biases. Taking both alleged shortcomings intoaccount in their own analyses, Miller and Sanjurjo report evidence insupport of the hot hand. It’s not clear whether this finding will bereplicated by other investigators, that the size of the effect is non-trivialand holds practical significance, or that it generalizes to other sports, but itmight be the case that sports momentum can hold some power in certaincircumstances.As this debate continues within and beyond the domain of academic

research, the open-minded skepticism of critical thinking will be essentialto arrive at accurate conclusions. It would be unwise either to accept orreject all claims for the power of sports momentum, let alone for potential



momentum effects beyond sports. For example, Pinker (2011) argues thattheMatthew effect, which takes its name from Biblical parables and is oftensummarized as “the rich get richer and the poor get poorer,” may applyacross societies: “everything seems to go right in some societies and wrongin others” (p. 608). Indeed, as is so often the case, the reality of momentumeffects may be nuanced. The uncritical thinking of System 1 will fall farshort of the scientific reasoning of a critical thinker in coming to under-stand these potential subtleties and complexities.

John Ruscio: How Critical Thinking Has Played an Important Rolein My Own Professional Career

Like many scientists, I try to be my own harshest critic by looking carefullyfor any weaknesses in my ideas, methods, or findings. Just as I was leavinggraduate school and beginning a job as an assistant professor, I wasperforming my first study using Paul Meehl’s taxometric method. Alongwith my wife and research collaborator, Ayelet Meron Ruscio, I was tryingto determine whether diagnosable depression was a categorical construct(i.e., individuals are either depressed or non-depressed) or a dimensionalconstruct (i.e., individuals vary along a continuum of depressive severity).I ran the taxometric analyses in the conventional manner and interpretedthe results in the usual way. The findings seemed clear, but I had somereservations. Though I admired the philosophical foundations of thetaxometric method, I found the procedural guidelines vague and theinterpretive standards subjective. I put our paper on hold while I spentabout six months designing and conducting a simulation study examiningtaxometric analyses under data conditions similar to ours. The resultsconfirmed my suspicions that the assumptions being made wheninterpreting taxometric results were in some ways oversimplified. When mywife and I revisited our findings in light of this new evidence, we completelyreversed our conclusion. Our report on this research was ultimatelypublished in the Journal of Abnormal Psychology. This helped both of ourcareers at this early stage, and it led to my most productive stream ofresearch over the next twenty years: how to reduce subjectivity in theimplementation of taxometric analysis and the interpretation of taxometricresults. What I learned along the way has helped me examine, refine, andeven develop other statistical methods. Most important have been thelessons that shaped my professional development. I learned firsthand thevalue of acquiring new skills to complete an investigation as rigorously aspossible, of taking the time to explore all doubts rather than rushing topublish initial conclusions, and of thinking critically about my own workthrough the deliberate, effortful, and slow means of scientific reasoning.



Kevin Brady: How Critical Thinking Has Played an ImportantRole in My Own Professional Career

Growing up as an avid sports fan, I often took what the television analystssaid as fact. After all, they should knowmore than any casual fan, and wouldhave more insight into what creates wins and losses than someone like me.But as I went through my studies as an undergraduate student majoring inpsychology, I began to realize how important critical thinking was andbrought this into my day-to-day life. This prompted great interest instudying between-game momentum in the NFL to find if there isa potentially gigantic hole in the way we cover, analyze, and predict NFLgames. Momentum is one of those generally-accepted principles which isn’toften submitted to critical thinking. The findings of my research projectonly increased my desire to think through issues critically, and to do so byscientifically questioning evidence. I am still very early in my professionalcareer, but I aim each day to be consciously skeptical and think critically.

Critical Thinking about Critical Thinking Questions

1. Dr. Flurpple is recognized as one of the most accomplished scientists of hergeneration, with an influential record of scholarly publications and animpressive list of professional honors and awards. At the same time, sheholdsmany superstitious beliefs. For example,Dr. Flurpple alwayswears herlucky earrings when delivering an important talk, and she refuses to stay onthe 13th floor of any hotel because she believes it brings bad luck. How doesan understanding of critical thinking as scientific reasoning that engagesSystem 2 help to explain howDr. Flurpple can be such a successful scientistwhile thinking uncritically about more mundane issues?

2. Suppose that NFL team A trades for a star player and wins theirnext several games, team B experiences an injury to a star playerand loses their next several games, and a sports analyst says thesestreaks demonstrate the effects of positive momentum (team A) andnegative momentum (team B). What are two plausible alternativeexplanations for these observed streaks in performance?

3. Cognitive heuristics such as availability, representativeness, oranchoring and insufficient adjustment can lead to systematic biasesin judgments or decisions. How is this different from saying thatthey can lead to random errors? Why is this distinction important?

4. Dr. Glurpple is familiar with the relevant empirical research and finds ithard to understand why so many players, coaches, fans, and analysts



believe in the power of sports momentum. If only people would thinkmore carefully about the alleged streaks in performance that they take assupportive evidence, he argues, they would know better. Aside fromrecognizing this as an instance of hindsight bias (the feeling that somethingseems obvious or inevitable after the results are known), what isDr. Glurpple overlooking with respect to the ability to think criticallyabout a phenomenon like sports momentum? In other words, is it suffi-cient to “think more carefully” to arrive at a more accurate conclusion?

5. Many academics believe there is a momentum effect in the careers ofworking scientists. Early success in securing external funding forresearch and publishing findings in leading scholarly journals is thoughtto predict later success, and early failure to predict later failure. In whatimportant ways is belief in the power of momentum in scientific careersdifferent from belief in power of sports momentum? How could theformer be tested?

Key Terms

Anchoring and insufficient adjustment Cognitive heuristic withwhich we estimate an unknown quantity by starting with an initialvalue and moving in the correct direction, but not far enough.

Availability Cognitive heuristic with which we estimate frequency orprobability according to the ease with which instances can be recalled.

Cognitive heuristic Mental shortcut that improves efficiency in reachingjudgments or decisions.

Representativeness Cognitive heuristic with which we estimateprobability or likelihood according to perceived similarity.

Sports momentum The positive or negative change in cognition, affect,physiology, and behavior caused by an event or series of events that affectseither the perceptions of the competitors or, perhaps, the quality ofperformance and the outcome of the competition. Positive momentum isassociated with periods of competition, such as a winning streak, in whicheverything seems to “go right” for the competitors. In contrast, negativemomentum is associated with periods, such as a losing streak, wheneverything seems to “go wrong”.



System 1 Effortless, fast, automatic thinking that takes place mostlyoutside conscious awareness.

System 2 Effortful, slow thinking that requires deliberate attention andcontrol.

REFERENCES

Avugos, S., Köppen, J., Czienskowski, U., Raab, M., & Bar-Eli, M. (2013). The“hot hand” reconsidered: A meta-analytic approach. Psychology of Sport andExercise, 14, 21–27. DOI:10.1016/j.psychsport.2012.07.005

Fry, M. J., & Shukairy, F. A. (2012). Searching for momentum in the NFL.Journal of Quantitative Analysis in Sports, 8, 1–20. DOI:10.1515/1559-0410.1362

Furnham, A., & Boo, H. C. (2011). A literature review of the anchoring effect. TheJournal of Socio-Economics, 40, 35–42. DOI:10.1016/j.socec.2010.10.008

Gandar, J., Zuber, R., O‘Brien, T., & Russo, B. (1988). Testing rationality in thepoint spread betting market. The Journal of Finance, 43, 995–1008. DOI:10.1111/j.1540-6261.1988.tb02617.x

Gilovich, T., Vallone, R., & Tversky, A. (1985). The hot hand in basketball: Onthe misperception of random sequences. Cognitive Psychology, 17, 295–314.DOI:10.1016/0010-0285(85)90010-6

Johnson, A. W., Stimpson, A. J., & Clark, T. K. (2012). Turning the tide: Bigplays and psychological momentum in the NFL. Sloan Sports AnalyticsConference Papers and Proceedings. March 2–3. Online. https://tinyurl.com/6ukyx36

Kent, M. (2006). Oxford dictionary of sports science and medicine (3rd ed.).New York: Oxford University Press.

Kahneman, D. (2011). Thinking, fast and slow. New York: Farrar, Straus, andGiroux.

Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment ofrepresentativeness. Cognitive Psychology, 3, 430–454. DOI:10.1007/978-94-010-2288-0_3

Lewis, M. (2003). Moneyball: The art of winning an unfair game. New York:Norton.

Miller, J. B., & Sanjurjo, S. (2016). A primer and frequently asked questions forsurprised by the gamblers and hot hand fallacies? A truth in the law of smallnumbers (Miller and Sanjurjo 2015). Online. DOI:10.2139/ssrn.2728151

Moskowitz, T., & Wertheim, L. J. (2011). Scorecasting: The hidden influencesbehind how sports are played and games are won. New York: Crown.

Nickerson, R. S. (2002). The production and perception of randomness.Psychological Review, 109, 330–357. DOI:10.1037/0033-295X.109.2.330

Pinker. S. (2011). The better angels of our nature: Why violence has declined.New York: Viking.



Sagan, C. (1995). The demon-haunted world: Science as a candle in the dark.New York: Random House.

Simon, H. A. (1956). Rational choice and the structure of the environment.Psychological Review, 63, 129–138. DOI:10.1037/h0042769

Tversky, A., & Kahneman, D. (1973). Availability: A heuristic for judging fre-quency and probability. Cognitive Psychology, 5, 207–232.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristicsand biases. Science, 185, 1124–1131. DOI:10.1126/science.185.4157.1124


critical thinking as scientiﬁc reasoning

Documents