2015 European Intelligence and Security Informatics Conference The Role of Cognitive Biases in Criminal Intelligence Analysis and Approaches for their Mitigation Eva-Catherine Hillemann, Alexander Nussbaumer, Dietrich Albert Knowledge Technologies Institute, Graz University of Technology Graz, Austria [email protected] principles of visual analytics. Thereby, a key focus of this project is on appreciating the importance of the human issues on the design of the VALCRI information exploitation system in the context of police intelligence analysis. These human issues are to deal with the legal, ethical and privacy implications of technology, mitigation of cognitive bias caused by such automated systems, how sense-making occurs in this context, and how information and knowledge should be structured to support the human reasoning process. Focusing on the human issues aspect of cognitive biases, the main research areas of interest involve i) the identification of relevant biases in this field, ii) developing strategies for their detection and mitigation, and iii) the integration of these findings in the VALCRI system by providing design guidelines that inform the technical design. This paper aims at giving an overview of the current research in the field of cognitive biases that serves the theoretical basis for being able to answer these research questions (Section 2). In addition to this, first results of the research endeavors described above are presented in Section 3 and Section 4. An outline of future work is given in Section 5. Abstract In the domain of criminal intelligence analysis, each day an analyst has to make sense and to create insight of a large amount of different data. However, due to the nature of human cognition, these cognitive processes may lead to systematic errors, so-called cognitive biases. In this paper, based on relevant stateof-the-art, preliminary ideas how to support the mitigation of cognitive biases - included in a visual analytics environment for criminal intelligence analysis currently being developed in the VALCRI project - are presented. By analysing user requirements and aligning them to the state-of-the-art research in the area of cognitive biases and their mitigation, eight cognitive biases have been identified as the most relevant ones. Six design guidelines have been proposed that help to mitigate one of them, the confirmation bias. At the current stage, the suggested mitigation strategies and resulting guidelines are the basis for further research, development, and experiments in order to derive evidence-based scientific results. Keywords—cognitive bias; bias mitigation strategies; design guidelines; criminal intellegince ananylsis I. INTRODUCTION In our technology-driven society, managing the flood of available data has become tremendously challenging. A criminal intelligence analyst has to quickly find relevant information from very large datasets and has to piece them together so that it is possible to draw a sensible, reasonable, justifiable, and defensible conclusion. Especially in the field of criminal intelligence analysis, making a right decision is essential as failures can have severe consequences such as wrongful convictions. However, sometimes the process of decision and sense-making is influenced by prior beliefs or experiences that may lead to wrong decisions. To understand such failures in human decision and sense-making, a lot of research especially in the field of cognitive psychology has been done. So-called cognitive biases have been proposed as possible explanation for such pitfalls in judgment and decision making [1]. In criminal intelligence analysis, the main challenge is to present data in such a way that such cognitive failures are avoided and consequently sense-making is supported. This is precisely the point where the European project VALCRI (www.valcri.org) comes into play. Addressing the challenges of todays’ law enforcement agencies, the main aim of this project is to support criminal intelligence analysts in their information exploitation processes – decision making and sense- making processes – by providing appropriate data analytic tools following the 978-1-4799-8657-6/15 $31.00 © 2015 IEEE DOI 10.1109/EISIC.2015.9 II. LITERATURE REVIEW To effectively manage information collection and processing as well as simultaneously avoiding to be overwhelmed by too much information, humans unconsciously apply “heuristics”. Such rule of thumbs or short cuts are primarily used to simplify a cognitive task in order to allow for making a decision when a fast decision is required and time and resources to make it in a rational manner is limited. Thus, they do not give optimal solutions, just “good enough” solutions that allow humans to save efforts and time but sometimes at the cost of accuracy [2][3]. These sometimes resulting “systematic errors” in judgment and decision making are referred to as cognitive biases. In the heuristics-and-bias program elaborated by [4] a cognitive bias is understood as “a pattern of deviation in judgment that occurs in particular situations, leading to perceptual distortion, inaccurate judgment, illogical interpretation, or what is broadly called irrationality”. Recent research on heuristics focuses more on the “fast and frugal” nature of heuristics as a means to reach “good enough” decisions in complex situations characterized by multiple decision criteria, high uncertainty, and time pressure [5][6]. Consequently, a selection of a specific heuristic is not necessarily the product of cognitive limitations 124 125 In the SIRIUS program of the Intelligence Advanced Research Project Activity (http://www.iarpa.gov/) next to the confirmation bias, five additional cognitive biases being relevant in the context of intelligence analysis have been examined. Fundamental Attribution Error (Attribution Bias, or Correspondence Bias, or Overattribution Effect) that can be described as the tendency to over-emphasize personality-based explanations for behaviors observed in others while underestimating the role of situational influences on the same behavior. The bias blind spot named after the visual blind spot that is the tendency to see the errors in another analyst’s collection and interpretation work while not seeing mistakes in their own research work [18]. Anchoring Bias or anchoring is the tendency to rely too heavily or "anchor" on a past reference or on one trait or piece of information when making decisions. The representativeness bias that leads to base rate fallacy or base rate neglect, is primarily used when making judgments about the probability of an event [1]. Base rate fallacy is the tendency to base judgments on specifics, ignoring general statistical information. In assessing a situation, numerical data summarizing information about similar case are available to the analyst. This numerical information is called base rate or prior probability [19]. In the context of (criminal) intelligence analysis only a small number of classification schemes and taxonomies has been suggested. One of the best-known taxonomy in this area is the one suggested and elaborated by [14]. Heuer’s classification system [14] is directly related to the activities of an intelligence analyst. It includes a collection of in sum 19 cognitive biases that can be categorized into four top-level categories: i) evaluation of evidence, ii) perception of cause and effect, iii) estimating probabilities, and iv) evaluation of intelligence reporting. Another good-known taxonomy is provided by [20]. The US Government [20] suggests the following biases as being relevant for the intelligence analysis process (based on [14]): i) perceptual biases including expectations, resistance, and ambiguities; ii) biases in evaluating evidence consisting of consistency, missing information, and discredited evidence; iii) biases in estimating probabilities with availability, anchoring, and overconfidence; and iv) biases in perceiving causality with rationality and attribution. but rather evoked by the characteristics of the environment. A cognitive bias in that sense is the tendency to solve problems using a particular heuristic. A vast body of literature has identified a vast amount of different biases and heuristics that play a significant role in the decision making and sense-making process. Having this large number of decision biases suggested by literature, some of the researchers have elaborated a taxonomy of biases, a classification scheme allowing for clarifying the relationship and influence of categories of biases on decision-making. The most influential taxonomy of cognitive biases is the taxonomy of [4] using the following three types of heuristics: availability (assessments based on one’s ability to recall past events), representativeness (assessments based on similarity between events), and adjustment and anchoring (assessments bounded by initial judgments). Remus and Kottemann [7] grouped heuristics based on their relevance to data presentation and information processing in the context of intelligent decision support systems. Hogarth [8] categorized heuristics based on the four stages in decision - making ranging from acquisition, processing, output, to feedback resulting in 33 biases. Arnott [9] extended Hogarth’s model of human judgment by grouping 37 biases obtained from a literature review according to their similarity that “emerges naturally” ([9], p.31). The resulting groupings are Memory, Statistical, Confidence, Adjustment, Presentation, and Situation Biases. When looking at the literature, there are a lot of other reviews of decision biases available that offer some kinds of taxonomies or classification schemes, for instance [10][11][12], as well as Wikipedia and RationalWiki and many more – mentioned here just for the sake of completeness. In the context of criminal intelligence analysis, analysts have to deal with a huge amount of different information in order to find new insights, make sense of these data and consequently to make valuable and sound decisions. Sense making in this context means that analysts have to find and interpret relevant facts by actively constructing a meaningful and functional representation of some aspects of the whole picture. This process of developing raw information into finished intelligence is described by the intelligence cycle [13], consisting of mainly five steps: i) Planning and Direction, ii) Collection, iii) Processing, iv) Production and Analysis, and v) Dissemination. Cognitive biases can occur in every phase of the intelligence cycle causing in errors of judging, such as discounting, misinterpreting, ignoring, rejection or overlooking information [14]. One of the most well-known cognitive biases in decision making and thus also in the context of (criminal) intelligence analysis is the Confirmation Bias, in which an analyst disproportionally considers and selects information that supports the initial expectation and hypothesis. In general, the idea that working hypotheses can enhance accuracy of diagnostic results and research is good [15], however one problem known from psychological research is that once having formed a hypothesis individuals tend to seek, interpret and create information to support their initial thoughts [16]. A bias that is closely related to the confirmation bias, is the backfire effect. This is the tendency to intensify former beliefs and assumptions after receiving disconfirming evidences or information [17]. III. IDENTIFYING COGNITIVE BIASES IN VALCRI The identification process is based on the bias categorization of a previous project (RECOBIA), an extensive literature review, and an end user requirements elicitation. By taking into account literature reviews and end user requirements, the bias identification process follows a theoretical and top-down approach, as well as an end usercentered bottom up approach. A. Cognitive Bias Classification in RECOBIA In the RECOBIA project (www.recobia.eu) a methodologically sound classification of cognitive biases has been elaborated. The major aim of this project supported by the European Commission was to improve the quality of intelligence analysis by reducing the negative impact of cognitive biases upon intelligence analysis. In a first step, based on an extensive literature review, a data set consisting of the description and definitions of in total 126 125 making and cognitive bias research in order to discover potential causes and effects on the sense-making process in terms of occurrence of cognitive biases. Take this user story as example: "As an analyst I want to have a tool that allows for filtering search results and other data sets (e.g. identify all burglaries and see their distribution)" - this activity "filtering search results" has been analyzed from a cognitive bias point of view - led by the following questions: How does this activity is impacted by biases? Which biases can occur when filtering search results? Which biases are relevant and can have an impact on the subsequent process of sense- and decision making? In the presented example, the confirmation bias can have an influence the way how a filter are set up by the analyst. When having a specific hypothesis in mind, the risk is that the analyst chooses those filters that are in line with his or her beliefs and confirm his initial hypothesis. After doing this analysis for each user story, in a second step, both experts have discussed the results and findings in small face-to-face meetings in order to identify the degree of agreement among them as raters and to ensure some homogeneity and consensus in the identification of relevant biases. 288 unique cognitive biases has been developed. In order to reduce this vast amount of different biases, a systematic classification has been done applying an algorithmic approach – the Formal Concept Analysis (FCA) [21]. The application of this approach allows for grouping similar cognitive biases that are basically the same into clusters. The main idea behind is, that when having such a grouping, the application of a mitigation strategy for one particular bias should also work for the other biases of the same group. After the application of FCA, 75 clusters of cognitive biases and heuristics has been identified. B. Relevant Biases from Literature Review When having a look at the research literature on cognitive biases, it becomes clear that there is a wide range and number of different biases that seems to be of relevance in the decision and sense-making process, in general, and in the context of criminal intelligence analysis, in particular. Classification systems seem to be based on subjective grouping without giving an understanding on their methodological basis. They often include a lot of cognitive biases that are sometimes listed more than once and that are strongly related to each other while others are completely missing. Furthermore, oftentimes the same biases are described with different terms by different authors although their meaning is the same. In the context of (criminal) intelligence analysis, those cognitive biases seem to be highly relevant that have been examined by [14] and [20]. In both classification schemes cognitive biases can be related to the following four categories: i) biases in evaluating evidence, ii) biases in perception of cause and effect, iii) biases in estimating probabilities, and iv) biases in the evaluation of intelligence reporting (perceptual biases). In the SIRIUS project, individual cognitive biases has been identified without assigning them to one of the before mentioned categories. More specifically, these biases are: Confirmation Bias, Fundamental Attribution Error, Bias Blind Spot, Anchoring Bias, Representativeness Bias, and Projection Bias. D. Relevant Cognitive Biases in VALCRI The outcome of the requirements analysis was aligned to the state-of-the-art research resulting in a preliminary set of biases that seems of relevance for the project based on their amount of occurrence within the user stories. In sum, the following eight cognitive biases have been identified as the most relevant ones: Confirmation Bias. The tendency to search for or interpret information in a way that confirms one's preconceptions or hypotheses; Anchoring and Adjustment Effect. The tendency to rely too heavily or "anchor" on a past reference or on one trait or piece of information when making decisions; Clustering Illusion. The tendency to see patterns where actually no patters exist; Framing Effect. The tendency to draw different conclusions from the same information, depending on how that information is presented; Availability heuristic. The tendency to make judgments about the probability of events occurring by how easily these events are brought to mind; Base Rate fallacy. The tendency to base judgments on specifics, ignoring general statistical information; Selective Perception. Selective perception occurs when people pay particular attention to some parts of their business environment to the point where it distorts the reality of the situation; Group Think. Groupthink leads to a deterioration of mental efficiency, reality testing and moral judgment resulting from group pressure; C. Cognitive Biases Identified in the Requirements Analysis With regard to the identification of cognitive biases that are of main interest in the context of VALCRI, a user-centered bottom-up approach has been applied focusing on the requirements of VALCRI’s end-users. The main purpose of this requirements analysis was to collect more generally formulated needs, expectations, opinions, and problems towards VALCRI’s user interface and functionalities. Requirements elicitation was carried out by project partners through interviews, workshops and other face-to-face meetings with end-users (intelligence analysts in law enforcement) in order to establish a comprehensive understanding of the end-users’ opinion and needs, tasks they need to accomplish, and probably their experiences and problems with existing techniques. The result of this work was a list of over 700 statements in the form of user stories that express requirements of the end users. A user story in our case is a statement that follows the format "As a <role>, I want <goal/desire> so that <benefit>". The exact procedure of identifying relevant biases in the different user stories was as follows: each user story was analyzed separately by two experts in the field of decision IV. DESIGN GUIDELINES FOR COGNITIVE BIAS MITIGATION In order to support the mitigation of a subset of the identified relevant cognitive biases, design guidelines have been developed that inform the system design. The goal of these guidelines is to give recommendations, so that the 127 126 findings, the future work will focus on experiments and further elaboration of the detection and mitigation strategies. system is being designed in a way that rather unconsciously reduces the occurrence of cognitive biases. Instead of making the analyst aware of biases and using prompts, the system should be designed that biases are mitigated due to the way of working induced by the system design. Though not implemented and evaluated, they are considered as a valuable basis for the system development and experiments. At this stage of the project, a main focus is on the confirmation bias and possibilities of its mitigation as it is doubtless one of the most influential cognitive biases in the field of decision and sense-making and in the context of criminal intelligence analysis. Thus, this section lists a preliminary list of guidelines developed for the purpose of mitigating the confirmation bias. Though not implemented and evaluated, we consider them as a valuable basis for the system development and experiments. ACKNOWLEDGMENT The research leading to these results in project VALCRI has received funding from the European Union 7th Framework Programme (FP7/2007-2013) under grant agreement no FP7IP-123456 REFERENCES [1] [2] [3] [4] Visualization types. Relevant information can be visualized through the use of different visualization techniques. The user interface should provide multiple options to visualize searched information. Levels of uncertainty. If the provided data is not complete or if data is not confirmed, an indication should be provided to make the user aware of it. This can be realized through the use of visualizations. They can be used to keep disconfirming evidence, probabilities of deception, and amounts of supporting information in front of the user to help to reduce the likelihood of this information being lost in data volume. Computerized critic questions. This strategy is also called the devil's advocate method. This procedure has the advantage that the analyst has to be prepared to defend his/her reasoning. Group decision making. When working or discussing in a group, everybody has to defend their own hypothesis and consider other (possibly contradicting) hypotheses. Feedback. If analysts think or analyze into wrong directions and the system can detect such failures, than feedback would help to rethink the current hypothesis. [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] V. CONCLUSIONS AND NEXT STEPS [15] In this paper, an overview of cognitive biases that are of relevance in the context of criminal intelligence analysis as well as the underlying identification process is given. Thus, the relevant state-of-the-art in the research area of cognitive biases in general and in the field of criminal intelligence analysis is presented. By analyzing user requirements gathered in the context of VALCRI and aligning them to the state-ofthe-art research, eight cognitive biases have been identified as most relevant (confirmation bias, anchoring and adjustment effect, clustering illusion, framing effect, availability heuristics, base rate fallacy, selective perception, and group think). Focusing on the confirmation bias, seven preliminary guidelines have been proposed how to build a system that helps to mitigate this bias. In order to base the concepts for bias detection and mitigation, and their technical application on solid research [16] [17] [18] [19] [20] [21] 128 127 D. Kahneman and A. Tversky, “Subjective probability: A judgment of representativeness,” Cognitive Psychology, vol. 3, pp.430–454, 1972. J.W. Payne, J.R. Bettman, and E.J. Johnson, The adaptive decision maker. New York: Cambridge University Press, 1993. A.K. Shah and D.M. Oppenheimer, “Easy does it: The role of fluency in cue weighting, “ 2007. A. Tversky and D. Kahneman, “Judgment under uncertainty: Heuristics and biases,” Science, vol. 185, pp. 1124–1131, 1974. G. Gigerenzer, “How to make cognitive illusions disappear: Beyond heuristics and biases,” Psychological Review, vol. 103, pp. 592-596, 1991. G. Gigerenzer, “On narrow norms and vague heuristics: a reply to Kahneman and Tversky,” European Review of Social Psychology, vol. 2, pp. 83–115, 1996. W.E. Remus and J.E. Kottemann, “Toward intelligent decision support systems: an artificially intelligent statistician,” MIS Quarterly, vol. 10, pp. 403–418, 1986. R. Hogarth, Judgement and Choice: The Psychology of Decision. John Wiley & Sons, New York, 1987. D. Arnott, “A Taxonomy of Decision Biases,” Technical Report 1/98, Monash University, School of Information Management and Systems, 1998. J. Baron, Thinking and deciding (4th ed.). New York: Cambridge University Press, 2007. C.R. Carter, L. Kaufmann, and A. Michel, “Behavioural supply management: a taxonomy of judgment and decision-making biases,” International Journal of Physical Distribution & Logistics Management, vol. 37 (8), pp. 631– 669, 2007. L. Virine and M. Trumper, “Project Decisions: The Art and Science,” 2007. L. Johnson, “Making the intelligence “Cycle” work”, International Journal of Intelligence and CounterIntelligence, vol. 1 (4), pp.1-23, 1986. R.J.Jr. Heuer, Psychology of intelligence analysis. Washington, DC: Central Intelligence Agency, Center for the Study of Intelligence, 1999. G.R. Norman, L.R. Brooks, C.L. Colle, and R.M. Hatala, “The benefit of diagnostic hypotheses in clinical reasoning: Experimental study of an instructional intervention for forward and backward reasoning,” Cognition and Instruction, vol. 17, pp. 433–448, 1999. R.S. Nickerson, “Confirmation bias: A ubiquitous phenomenon in many guises,” Review of General Psychology, vol. 2, pp. 175–220, 1998. J.D. Lieberman and J. Arndt, “Understanding the limits of limiting instruction: Social psychological explanations for the failure of instructions to disregard pretrial publicity and other inadmissible evidence,” Psychology, Public Policy, and Law, vol. 6, pp. 677-711, 2000. E. Pronin, C. Puccio, and L. Ross, “36. Understanding Misunderstanding: Social Psychological Perspectives,” 2002. R.E. Nisbett, E. Borgida, R. Crandall, and H. Reed, “Popular induction: Information is not necessarily informative,” 1976. US Governement, A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis. Center for the Study of Intelligence, 2009. R. Wille, “Restructuring lattice theory: an approach based on hierarchies of concepts,” in Ordered sets, I. Rival, Ed. Dordrecht-Boston: Reidel, 1982, pp. 445–470.
