Uploaded by User8973

Cognitive Biases2015

2015 European Intelligence and Security Informatics Conference
The Role of Cognitive Biases in Criminal Intelligence
Analysis and Approaches for their Mitigation
Eva-Catherine Hillemann, Alexander Nussbaumer, Dietrich Albert
Knowledge Technologies Institute,
Graz University of Technology
Graz, Austria
[email protected]
principles of visual analytics. Thereby, a key focus of this
project is on appreciating the importance of the human issues
on the design of the VALCRI information exploitation system
in the context of police intelligence analysis. These human
issues are to deal with the legal, ethical and privacy
implications of technology, mitigation of cognitive bias
caused by such automated systems, how sense-making occurs
in this context, and how information and knowledge should be
structured to support the human reasoning process.
Focusing on the human issues aspect of cognitive biases,
the main research areas of interest involve i) the identification
of relevant biases in this field, ii) developing strategies for
their detection and mitigation, and iii) the integration of these
findings in the VALCRI system by providing design
guidelines that inform the technical design.
This paper aims at giving an overview of the current
research in the field of cognitive biases that serves the
theoretical basis for being able to answer these research
questions (Section 2). In addition to this, first results of the
research endeavors described above are presented in Section 3
and Section 4. An outline of future work is given in Section 5.
Abstract In the domain of criminal intelligence analysis, each day
an analyst has to make sense and to create insight of a large
amount of different data. However, due to the nature of human
cognition, these cognitive processes may lead to systematic errors,
so-called cognitive biases. In this paper, based on relevant stateof-the-art, preliminary ideas how to support the mitigation of
cognitive biases - included in a visual analytics environment for
criminal intelligence analysis currently being developed in the
VALCRI project - are presented. By analysing user requirements
and aligning them to the state-of-the-art research in the area of
cognitive biases and their mitigation, eight cognitive biases have
been identified as the most relevant ones. Six design guidelines
have been proposed that help to mitigate one of them, the
confirmation bias. At the current stage, the suggested mitigation
strategies and resulting guidelines are the basis for further
research, development, and experiments in order to derive
evidence-based scientific results.
Keywords—cognitive bias; bias mitigation strategies; design
guidelines; criminal intellegince ananylsis
In our technology-driven society, managing the flood of
available data has become tremendously challenging. A
criminal intelligence analyst has to quickly find relevant
information from very large datasets and has to piece them
together so that it is possible to draw a sensible, reasonable,
justifiable, and defensible conclusion. Especially in the field
of criminal intelligence analysis, making a right decision is
essential as failures can have severe consequences such as
wrongful convictions. However, sometimes the process of
decision and sense-making is influenced by prior beliefs or
experiences that may lead to wrong decisions. To understand
such failures in human decision and sense-making, a lot of
research especially in the field of cognitive psychology has
been done. So-called cognitive biases have been proposed as
possible explanation for such pitfalls in judgment and decision
making [1]. In criminal intelligence analysis, the main
challenge is to present data in such a way that such cognitive
failures are avoided and consequently sense-making is
supported. This is precisely the point where the European
project VALCRI (www.valcri.org) comes into play.
Addressing the challenges of todays’ law enforcement
agencies, the main aim of this project is to support criminal
intelligence analysts in their information exploitation
processes – decision making and sense- making processes – by
providing appropriate data analytic tools following the
978-1-4799-8657-6/15 $31.00 © 2015 IEEE
DOI 10.1109/EISIC.2015.9
To effectively manage information collection and
processing as well as simultaneously avoiding to be
overwhelmed by too much information, humans
unconsciously apply “heuristics”. Such rule of thumbs or short
cuts are primarily used to simplify a cognitive task in order to
allow for making a decision when a fast decision is required
and time and resources to make it in a rational manner is
limited. Thus, they do not give optimal solutions, just “good
enough” solutions that allow humans to save efforts and time
but sometimes at the cost of accuracy [2][3]. These sometimes
resulting “systematic errors” in judgment and decision making
are referred to as cognitive biases. In the heuristics-and-bias
program elaborated by [4] a cognitive bias is understood as “a
pattern of deviation in judgment that occurs in particular
situations, leading to perceptual distortion, inaccurate
judgment, illogical interpretation, or what is broadly called
irrationality”. Recent research on heuristics focuses more on
the “fast and frugal” nature of heuristics as a means to reach
“good enough” decisions in complex situations characterized
by multiple decision criteria, high uncertainty, and time
pressure [5][6]. Consequently, a selection of a specific
heuristic is not necessarily the product of cognitive limitations
In the SIRIUS program of the Intelligence Advanced
Research Project Activity (http://www.iarpa.gov/) next to the
confirmation bias, five additional cognitive biases being
relevant in the context of intelligence analysis have been
examined. Fundamental Attribution Error (Attribution Bias,
or Correspondence Bias, or Overattribution Effect) that can be
described as the tendency to over-emphasize personality-based
explanations for behaviors observed in others while
underestimating the role of situational influences on the same
behavior. The bias blind spot named after the visual blind spot
that is the tendency to see the errors in another analyst’s
collection and interpretation work while not seeing mistakes in
their own research work [18]. Anchoring Bias or anchoring is
the tendency to rely too heavily or "anchor" on a past
reference or on one trait or piece of information when making
decisions. The representativeness bias that leads to base rate
fallacy or base rate neglect, is primarily used when making
judgments about the probability of an event [1]. Base rate
fallacy is the tendency to base judgments on specifics,
ignoring general statistical information. In assessing a
situation, numerical data summarizing information about
similar case are available to the analyst. This numerical
information is called base rate or prior probability [19].
In the context of (criminal) intelligence analysis only a
small number of classification schemes and taxonomies has
been suggested. One of the best-known taxonomy in this area
is the one suggested and elaborated by [14]. Heuer’s
classification system [14] is directly related to the activities of
an intelligence analyst. It includes a collection of in sum 19
cognitive biases that can be categorized into four top-level
categories: i) evaluation of evidence, ii) perception of cause
and effect, iii) estimating probabilities, and iv) evaluation of
intelligence reporting.
Another good-known taxonomy is provided by [20]. The
US Government [20] suggests the following biases as being
relevant for the intelligence analysis process (based on [14]):
i) perceptual biases including expectations, resistance, and
ambiguities; ii) biases in evaluating evidence consisting of
consistency, missing information, and discredited evidence;
iii) biases in estimating probabilities with availability,
anchoring, and overconfidence; and iv) biases in perceiving
causality with rationality and attribution.
but rather evoked by the characteristics of the environment. A
cognitive bias in that sense is the tendency to solve problems
using a particular heuristic.
A vast body of literature has identified a vast amount of
different biases and heuristics that play a significant role in the
decision making and sense-making process. Having this large
number of decision biases suggested by literature, some of the
researchers have elaborated a taxonomy of biases, a
classification scheme allowing for clarifying the relationship
and influence of categories of biases on decision-making. The
most influential taxonomy of cognitive biases is the taxonomy
of [4] using the following three types of heuristics: availability
(assessments based on one’s ability to recall past events),
representativeness (assessments based on similarity between
events), and adjustment and anchoring (assessments bounded
by initial judgments). Remus and Kottemann [7] grouped
heuristics based on their relevance to data presentation and
information processing in the context of intelligent decision
support systems. Hogarth [8] categorized heuristics based on
the four stages in decision - making ranging from acquisition,
processing, output, to feedback resulting in 33 biases. Arnott
[9] extended Hogarth’s model of human judgment by
grouping 37 biases obtained from a literature review according
to their similarity that “emerges naturally” ([9], p.31). The
resulting groupings are Memory, Statistical, Confidence,
Adjustment, Presentation, and Situation Biases. When looking
at the literature, there are a lot of other reviews of decision
biases available that offer some kinds of taxonomies or
classification schemes, for instance [10][11][12], as well as
Wikipedia and RationalWiki and many more – mentioned here
just for the sake of completeness.
In the context of criminal intelligence analysis, analysts
have to deal with a huge amount of different information in
order to find new insights, make sense of these data and
consequently to make valuable and sound decisions. Sense
making in this context means that analysts have to find and
interpret relevant facts by actively constructing a meaningful
and functional representation of some aspects of the whole
picture. This process of developing raw information into
finished intelligence is described by the intelligence cycle
[13], consisting of mainly five steps: i) Planning and
Direction, ii) Collection, iii) Processing, iv) Production and
Analysis, and v) Dissemination. Cognitive biases can occur in
every phase of the intelligence cycle causing in errors of
judging, such as discounting, misinterpreting, ignoring,
rejection or overlooking information [14].
One of the most well-known cognitive biases in decision
making and thus also in the context of (criminal) intelligence
analysis is the Confirmation Bias, in which an analyst
disproportionally considers and selects information that
supports the initial expectation and hypothesis. In general, the
idea that working hypotheses can enhance accuracy of
diagnostic results and research is good [15], however one
problem known from psychological research is that once
having formed a hypothesis individuals tend to seek, interpret
and create information to support their initial thoughts [16]. A
bias that is closely related to the confirmation bias, is the
backfire effect. This is the tendency to intensify former beliefs
and assumptions after receiving disconfirming evidences or
information [17].
The identification process is based on the bias
categorization of a previous project (RECOBIA), an extensive
literature review, and an end user requirements elicitation. By
taking into account literature reviews and end user
requirements, the bias identification process follows a
theoretical and top-down approach, as well as an end usercentered bottom up approach.
A. Cognitive Bias Classification in RECOBIA
In the RECOBIA project (www.recobia.eu) a
methodologically sound classification of cognitive biases has
been elaborated. The major aim of this project supported by
the European Commission was to improve the quality of
intelligence analysis by reducing the negative impact of
cognitive biases upon intelligence analysis.
In a first step, based on an extensive literature review, a
data set consisting of the description and definitions of in total
making and cognitive bias research in order to discover
potential causes and effects on the sense-making process in
terms of occurrence of cognitive biases. Take this user story as
example: "As an analyst I want to have a tool that allows for
filtering search results and other data sets (e.g. identify all
burglaries and see their distribution)" - this activity "filtering
search results" has been analyzed from a cognitive bias point
of view - led by the following questions: How does this
activity is impacted by biases? Which biases can occur when
filtering search results? Which biases are relevant and can
have an impact on the subsequent process of sense- and
decision making? In the presented example, the confirmation
bias can have an influence the way how a filter are set up by
the analyst. When having a specific hypothesis in mind, the
risk is that the analyst chooses those filters that are in line with
his or her beliefs and confirm his initial hypothesis.
After doing this analysis for each user story, in a second
step, both experts have discussed the results and findings in
small face-to-face meetings in order to identify the degree of
agreement among them as raters and to ensure some
homogeneity and consensus in the identification of relevant
288 unique cognitive biases has been developed. In order to
reduce this vast amount of different biases, a systematic
classification has been done applying an algorithmic approach
– the Formal Concept Analysis (FCA) [21]. The application of
this approach allows for grouping similar cognitive biases that
are basically the same into clusters. The main idea behind is,
that when having such a grouping, the application of a
mitigation strategy for one particular bias should also work for
the other biases of the same group. After the application of
FCA, 75 clusters of cognitive biases and heuristics has been
B. Relevant Biases from Literature Review
When having a look at the research literature on cognitive
biases, it becomes clear that there is a wide range and number
of different biases that seems to be of relevance in the decision
and sense-making process, in general, and in the context of
criminal intelligence analysis, in particular. Classification
systems seem to be based on subjective grouping without
giving an understanding on their methodological basis. They
often include a lot of cognitive biases that are sometimes listed
more than once and that are strongly related to each other
while others are completely missing. Furthermore, oftentimes
the same biases are described with different terms by different
authors although their meaning is the same.
In the context of (criminal) intelligence analysis, those
cognitive biases seem to be highly relevant that have been
examined by [14] and [20]. In both classification schemes
cognitive biases can be related to the following four
categories: i) biases in evaluating evidence, ii) biases in
perception of cause and effect, iii) biases in estimating
probabilities, and iv) biases in the evaluation of intelligence
reporting (perceptual biases). In the SIRIUS project,
individual cognitive biases has been identified without
assigning them to one of the before mentioned categories.
More specifically, these biases are: Confirmation Bias,
Fundamental Attribution Error, Bias Blind Spot, Anchoring
Bias, Representativeness Bias, and Projection Bias.
D. Relevant Cognitive Biases in VALCRI
The outcome of the requirements analysis was aligned to
the state-of-the-art research resulting in a preliminary set of
biases that seems of relevance for the project based on their
amount of occurrence within the user stories.
In sum, the following eight cognitive biases have been
identified as the most relevant ones:
Confirmation Bias. The tendency to search for or interpret
information in a way that confirms one's preconceptions or
Anchoring and Adjustment Effect. The tendency to rely too
heavily or "anchor" on a past reference or on one trait or piece
of information when making decisions;
Clustering Illusion. The tendency to see patterns where
actually no patters exist;
Framing Effect. The tendency to draw different
conclusions from the same information, depending on how
that information is presented;
Availability heuristic. The tendency to make judgments
about the probability of events occurring by how easily these
events are brought to mind;
Base Rate fallacy. The tendency to base judgments on
specifics, ignoring general statistical information;
Selective Perception. Selective perception occurs when
people pay particular attention to some parts of their business
environment to the point where it distorts the reality of the
Group Think. Groupthink leads to a deterioration of mental
efficiency, reality testing and moral judgment resulting from
group pressure;
C. Cognitive Biases Identified in the Requirements Analysis
With regard to the identification of cognitive biases that
are of main interest in the context of VALCRI, a user-centered
bottom-up approach has been applied focusing on the
requirements of VALCRI’s end-users. The main purpose of
this requirements analysis was to collect more generally
formulated needs, expectations, opinions, and problems
towards VALCRI’s user interface and functionalities.
Requirements elicitation was carried out by project partners
through interviews, workshops and other face-to-face
meetings with end-users (intelligence analysts in law
enforcement) in order to establish a comprehensive
understanding of the end-users’ opinion and needs, tasks they
need to accomplish, and probably their experiences and
problems with existing techniques. The result of this work was
a list of over 700 statements in the form of user stories that
express requirements of the end users. A user story in our case
is a statement that follows the format "As a <role>, I want
<goal/desire> so that <benefit>".
The exact procedure of identifying relevant biases in the
different user stories was as follows: each user story was
analyzed separately by two experts in the field of decision
In order to support the mitigation of a subset of the
identified relevant cognitive biases, design guidelines have
been developed that inform the system design. The goal of
these guidelines is to give recommendations, so that the
findings, the future work will focus on experiments and
further elaboration of the detection and mitigation strategies.
system is being designed in a way that rather unconsciously
reduces the occurrence of cognitive biases. Instead of making
the analyst aware of biases and using prompts, the system
should be designed that biases are mitigated due to the way of
working induced by the system design. Though not
implemented and evaluated, they are considered as a valuable
basis for the system development and experiments.
At this stage of the project, a main focus is on the
confirmation bias and possibilities of its mitigation as it is
doubtless one of the most influential cognitive biases in the
field of decision and sense-making and in the context of
criminal intelligence analysis. Thus, this section lists a
preliminary list of guidelines developed for the purpose of
mitigating the confirmation bias. Though not implemented and
evaluated, we consider them as a valuable basis for the system
development and experiments.
The research leading to these results in project VALCRI has
received funding from the European Union 7th Framework
Programme (FP7/2007-2013) under grant agreement no FP7IP-123456
Visualization types. Relevant information can be
visualized through the use of different visualization
techniques. The user interface should provide multiple options
to visualize searched information.
Levels of uncertainty. If the provided data is not
complete or if data is not confirmed, an indication should be
provided to make the user aware of it. This can be realized
through the use of visualizations. They can be used to keep
disconfirming evidence, probabilities of deception, and
amounts of supporting information in front of the user to help
to reduce the likelihood of this information being lost in data
Computerized critic questions. This strategy is also
called the devil's advocate method. This procedure has the
advantage that the analyst has to be prepared to defend his/her
Group decision making. When working or discussing
in a group, everybody has to defend their own hypothesis and
consider other (possibly contradicting) hypotheses.
Feedback. If analysts think or analyze into wrong
directions and the system can detect such failures, than
feedback would help to rethink the current hypothesis.
In this paper, an overview of cognitive biases that are of
relevance in the context of criminal intelligence analysis as
well as the underlying identification process is given. Thus,
the relevant state-of-the-art in the research area of cognitive
biases in general and in the field of criminal intelligence
analysis is presented. By analyzing user requirements gathered
in the context of VALCRI and aligning them to the state-ofthe-art research, eight cognitive biases have been identified as
most relevant (confirmation bias, anchoring and adjustment
effect, clustering illusion, framing effect, availability
heuristics, base rate fallacy, selective perception, and group
think). Focusing on the confirmation bias, seven preliminary
guidelines have been proposed how to build a system that
helps to mitigate this bias.
In order to base the concepts for bias detection and
mitigation, and their technical application on solid research
D. Kahneman and A. Tversky, “Subjective probability: A judgment of
representativeness,” Cognitive Psychology, vol. 3, pp.430–454, 1972.
J.W. Payne, J.R. Bettman, and E.J. Johnson, The adaptive decision
maker. New York: Cambridge University Press, 1993.
A.K. Shah and D.M. Oppenheimer, “Easy does it: The role of fluency in
cue weighting, “ 2007.
A. Tversky and D. Kahneman, “Judgment under uncertainty: Heuristics
and biases,” Science, vol. 185, pp. 1124–1131, 1974.
G. Gigerenzer, “How to make cognitive illusions disappear: Beyond
heuristics and biases,” Psychological Review, vol. 103, pp. 592-596,
G. Gigerenzer, “On narrow norms and vague heuristics: a reply to
Kahneman and Tversky,” European Review of Social Psychology, vol.
2, pp. 83–115, 1996.
W.E. Remus and J.E. Kottemann, “Toward intelligent decision support
systems: an artificially intelligent statistician,” MIS Quarterly, vol. 10,
pp. 403–418, 1986.
R. Hogarth, Judgement and Choice: The Psychology of Decision. John
Wiley & Sons, New York, 1987.
D. Arnott, “A Taxonomy of Decision Biases,” Technical Report 1/98,
Monash University, School of Information Management and Systems,
J. Baron, Thinking and deciding (4th ed.). New York: Cambridge
University Press, 2007.
C.R. Carter, L. Kaufmann, and A. Michel, “Behavioural supply
management: a taxonomy of judgment and decision-making biases,”
International Journal of Physical Distribution & Logistics Management,
vol. 37 (8), pp. 631– 669, 2007.
L. Virine and M. Trumper, “Project Decisions: The Art and Science,”
L. Johnson, “Making the intelligence “Cycle” work”, International
Journal of Intelligence and CounterIntelligence, vol. 1 (4), pp.1-23,
R.J.Jr. Heuer, Psychology of intelligence analysis. Washington, DC:
Central Intelligence Agency, Center for the Study of Intelligence, 1999.
G.R. Norman, L.R. Brooks, C.L. Colle, and R.M. Hatala, “The benefit of
diagnostic hypotheses in clinical reasoning: Experimental study of an
instructional intervention for forward and backward reasoning,”
Cognition and Instruction, vol. 17, pp. 433–448, 1999.
R.S. Nickerson, “Confirmation bias: A ubiquitous phenomenon in many
guises,” Review of General Psychology, vol. 2, pp. 175–220, 1998.
J.D. Lieberman and J. Arndt, “Understanding the limits of limiting
instruction: Social psychological explanations for the failure of
instructions to disregard pretrial publicity and other inadmissible
evidence,” Psychology, Public Policy, and Law, vol. 6, pp. 677-711,
E. Pronin, C. Puccio, and L. Ross, “36. Understanding
Misunderstanding: Social Psychological Perspectives,” 2002.
R.E. Nisbett, E. Borgida, R. Crandall, and H. Reed, “Popular induction:
Information is not necessarily informative,” 1976.
US Governement, A Tradecraft Primer: Structured Analytic Techniques
for Improving Intelligence Analysis. Center for the Study of
Intelligence, 2009.
R. Wille, “Restructuring lattice theory: an approach based on hierarchies
of concepts,” in Ordered sets, I. Rival, Ed. Dordrecht-Boston: Reidel,
1982, pp. 445–470.