A human-in-the-loop approach to understanding situation awareness in cyber defence analysis

In this paper we argue for a human-in-the-loop approach to the study of situation awareness in computer defence analysis (CDA). The cognitive phenomenon of situation awareness (SA) has received significant attention in cybersecurity/CDA research. Yet little of this work has attended to the cognitive aspects of situation awareness in the CDA context; instead, the human operator has been treated as an abstraction within the larger human-technology system. A more human-centric approach that seeks to understand the socio-cognitive work of human operators as they perform CDA will yield greater insights into the design of tools and interfaces for CDA. As support for this argument, we present our own work employing the Living Lab Framework through which we ground our experimental findings in contextual knowledge of real-world practice.


Introduction
Situation awareness (SA) has received widespread attention in research on cyber defence analysis. In particular, Endsley's (1995aEndsley's ( , 1995b '3 Level' model of SA has received the most attention among cyber-security scholars; primarily serving as a conceptual basis for measuring system performance. Indeed, much of the research on SA in cybersecurity, or cyber-SA, has primarily taken an algorithmic perspective (Orlikowski & Iacono, 2001); focusing primarily on the automation and the development of new defensive tools for protection, detection and response . Examples of this work include data visualizations (D'Amico, A. & Larkin, 2001), data fusion methods for tracking cyber-attacks (Stotz & Sudit, 2007;Yang, Shanchieh J., Stotz, Holsopple, Sudit, & Kuhl, 2009), identification of internal and external threats using * Corresponding author e-mail: mjt241@smeal.psu.edu intelligent agents (Buford, Lewis, & Jakobson, 2008;Yen et al., 2010), and the use of probabilistic models to assess network vulnerability (Peng, Li, Xinming, Peng, & Levy, 2010;Tadda, G., Salerno, Boulware, Hinman, & Gorton, 2006). Although valuable, this body of work overlooks perhaps the most crucial component of cyber defence analysis: the human component (Boyce et al., 2011;Goodall, Lutters, & Komlodi, 2004). Indeed, much of the research on cyber-SA has paid little attention to how operators perform with existing technologies let alone whether or not these new technologies actually improve SA in human operators.
In this paper we argue for a human-in-the-loop perspective on cyber-SA; thus shifting analytical attention away from the development of new technologies towards the socio-cognitive work of human cyber security professionals. We believe that such a shift is critically necessary because human operators are central and critical to any cyber defence system and yet our understanding of how human operators cognitively operate in the unique environment of cyberspace remains poorly understood.
Our argument proceeds as follows. We begin with a discussion of situation awareness theory and its application to the cyber defence domain. Here we draw attention to the strengths and weaknesses of the different theoretical perspectives on SA as applied to cyber defence analysis. We then proceed to a discussion of the different methodological approaches to the study of cyber-SA with particular attention given to the application of the Living Lab Framework (LLF) to the study of cyber defence analysis. Finally, we conclude the paper with a discussion of our on-going research of cyber defence analysis using LLF. In this section we present our findings from our field research and show how those findings are informing our on-going laboratory-based experiments using scaledworld simulations.

Situation Awareness, Cyber Defense Analysis & Cyber-SA
An area of cognitive science that has received significant attention in CDA research is situation awareness. In this section we provide an overview of SA theory and its application to the domain of cyber defence analysis. In doing so we make two main arguments: (1) that extant research on cyber-SA has largely overlooked the human operator; and (2) that a distributed theory of SA may be more ideally suited to the study of cyber-SA than the predominantly used cognitive perspective.

Situation Awareness
Situation awareness is, in lay terms, the knowing of what is going on around you. In cognitive terms, one's SA is the degree to which one perceives task-salient cues in the environment, correctly understands the meaning of those cues, and is able to take the correct action to achieve a specific future state (Endsley, 1995b). The greater an individual has SA, the more likely the individual is to take the appropriate action; conversely, a lack of SA is a key factor in the commitment of errors by human operators (Endsley, 2000).
There are numerous high-profile examples of a lack of SA contributing to catastrophic failures including the crash of Air France Flight 447 as a result of pilots incorrectly reacting to faulty sensor data (Wise, 2012), and the partial meltdown of Reactor 2 at Three Mile Island as a result of plant operators failing to correctly understand the operating state of the reactor tower's cooling system (Perrow, 1999). Scholars have studied situation awareness in numerous domains such as aircraft piloting (Endsley, 1993;Endsley, Farley, Jones, Midkiff, & Hansman, 1998), air traffic control (Endsley & Rodgers, 1994), ship navigation (Lee & Sanquist, 2000), emergency response (Blandford & William Wong, 2004;McGrath & McGrath, 2005), C4i systems (French & Hutchinson, 2002;Salmon, P., Stanton, Walker, & Green, 2006), and surgical teams (Bardram, Hansen, & Soegard, 2006;Hazlehurst, McMullen, & Gorman, 2007). The cognitive perspective, rooted in Endsley's (1995bEndsley's ( , 2000 information-processing model of SA, is the most widely adopted theoretical perspective in SA research. Known as the "3 Level" model, the cognitive perspective sees SA as a human internal cognitive state comprised of perception (Level 1), comprehension (Level 2), and projection (Level 3). The analytical focus of the cognitive perspective is on the human operator's understanding of the environment at a particular point in time which is assessed using freeze-probe measurement techniques such as the Situation Awareness Global Assessment Technique (SAGAT) (Endsley, 1995a). One strength of the cognitive perspective is that it lends itself to the quantitative measurement of SA: for example we can compare a an individual's ability to detect salient cues in the environment against a known ground truth (Salmon, P. M. et al., 2008). A criticism of the cognitive perspective is that it does not scale well to other levels of analysis. For example, Stanton et al. (2009) argue that team SA is more than just the sum of individual team members' SA as those who have adopted the cognitive perspective have suggested. Additionally, the cognitive perspective is not well suited to capturing the ways that artefacts offload elements of SA because it assumes SA as a strictly psychological phenomenon.
The technological perspective argues that SA is instantiated in the presentation of information by a technological artefact. Implicit in this view is that SA resides within the artefact itself, typically in the form of information (Stanton, Neville A. et al., 2009). For example, the display of route information, travel time, traffic conditions, weather and fuel efficiency in a GPS navigational appliance is considered to be a display of the SA (information) held by the device.
The analytical focus of the technological perspective is the design and configuration of information presentations to most effectively convey the SA contained within the device. The utility of the technological perspective is that it accounts for cognitive artefacts and how best to display the information they contain. A flaw in the technological perspective is the assumption that information in itself is situation awareness, and that by providing access to that information, human SA is necessarily improved. As the Air France disaster makes abundantly clear, even when technological artefacts are conveying their 'awareness' human situation awareness may not improve and may even deteriorate (Wise, 2012).
The distributed perspective of SA is a hybrid theory that posits that SA resides in both human and technological agents; distributed throughout a sociotechnical system (Stanton, N. A. et al., 2006). Developed relatively recently, distributed situation awareness (DSA) theory seeks to integrate SA with distributed cognition. Within the broader system, different agents may have different SA, and the degree to which agents within the system share SA is a function of the extent to which their goals overlap. The analytical focus of the distributed perspective is the socio-cognitive system and interactions between agents within the system and the system, and agents and the environment. A strength of the distributed perspective is that it accounts for both human and technological SA; potentially providing a more valid description of the ways in which SA occurs in task environments in which technology is central to the task. Additionally, by moving away from the view that team SA is the aggregate SA of all individual team members, the distributed perspective likely gives a more accurate picture of distributed collaborative work. It is not clear, however, how to measure SA distributed across a sociotechnical system. Gorman, Cooke and Winner (Gorman, Cooke, & Winner, 2006) have argued for measuring the extent to team members' actions, comments, behaviours and interactions are coordinated. It has also been suggested that performance is a useful proxy measure of DSA on the assumption that greater SA results in greater performance (Salmon, P. et al., 2006;Stanton, N. A. et al., 2006;Walker et al., 2006).
Each theoretical perspective has its strengths and weaknesses, particularly in their application to the study cyber-situation awareness. As we will show below, a review of the extant literature reveals that the cognitive perspective is the predominant perspective espoused in cyber-SA research. What will also be shown is that scholars have, in practice, used elements of the cognitive perspective to measure the 'SA' of the technological artefact.

Cyber Defence Analysis & Cyber-SA
A review of the extant literature on situation awareness in CDA reveals that though cyber defence scholars have relied on the cognitive perspective as a basis for their research almost exclusively, the research is, in practice employing the technological perspective of SA (Mathew, Shah, & Upadhyaya, 2005;Okolica, McDonald, Peterson, Mills, & Haas, 2009;Yang, S. J., Byers, Holsopple, Argauer, & Fava, 2008;Yen et al., 2010). The notable exception is Tadda and Salerno's (2010) Situation Awareness Reference Model. The extent to which cyber-SA has been considered beyond the 3-Level model has typically been limited to discussions of what phenomena an analyst or system must perceive (c.f., Barford et al., 2010) or factors that make developing cyber-SA difficult (c.f., Yang, S. J. et al., 2008).
We identify two primary reasons for why SA has been applied to CDA research in this way. One, the levels of the Endsley model of SA closely align to the levels of the JDL Data Fusion Process Model that informs much of the CDA research. The Joint Directors of Laboratories (JDL) Data Fusion Process Model is a model that describes how disparate pieces of data detected by multiple sensors are fused into a coherent picture. In the JDL model, there are four levels of data fusion (Llinas & Hall, 1998). The first three, consisting of Object Refinement (Level 1), Situation Refinement (Level 2), and Threat Refinement (Level 3), are conceptually similar to the three levels of SA respectively (perception, comprehension, and detection). This close conceptual alignment provides a natural point on which to connect the work on data fusion that comprises much of the CDA research to the desired state of situation awareness.
A second reason for why CDA scholars have applied a limited form of SA theory to their research is that their research really is not about SA at all, but about the development of new technologies, techniques, and representations of information. The implicit assumption of this stream of research is that improved sensor/intelligent agent performance or improved presentation of cybersecurity related information will produce improved SA in human operators.
Although valuable, this research is limited in how much it informs our understanding of SA in computer defence analysis for three reasons. First, this stream of research tends to focus on intrusion detection exclusively when computer defence in practice consists of a variety of activities including policy, forensics, remediation, and administration.
Second, focusing on technology development does not tell us if and how the cognitive process of forming SA in cyber defence analysis may or may not be different as a result of the unique environmental properties of working in cyberspace. Finally, this stream of research treats the human operator as an abstraction. Little to no attention is given to how cyber defence analysts actually work; make use of information, and whether the tools they are developing actually improve their situation awareness.

A Human-Centric Approach to Cyber Defence Research
In order to better understand how computer defence analysts develop situation awareness it is both important to understand the cognitive processes of cyber-SA and the socio-cognitive work of CDA. To do so, we have adopted the Living Laboratory Framework to guide our data collection and analysis. The Living Lab Framework is detailed below; followed by a discussion of findings from our initial round of field work and a description of our ongoing experiments using scaled-world simulations.

The Living Laboratory Framework
We employ the Living Laboratory Framework (LLF) to guide our data collection and analysis. As an ecological approach to the study of cognitive work in multi-operator environments that combines both quantitative and qualitative analysis (McNeese, 1996), the LLF is well suited to the study of the socio-cognitive work of CDA for four reasons. First, the analytical focus of the LLF is on cognitive work by human agents, precisely what we are focusing on in this research. Second, the underlying premise of the framework is cognitive work is situated, jointly determined by agents and environments, and is often 'goal-directed, self-organizing, and intentional (McNeese, Perusich, & Rentsch, 2000).' This premise is an accurate description of CDA in practice. Third, the LLF provides a process for the kind of empirically based theory development of the type we seek to accomplish with this research. Finally, the multi-method structure of the LLF facilitates both methodological and participant triangulation thereby increasing the validity of findings.
The components are performed in a mutually-informing, iterative loop.
Ethnographic observation is used to capture cognition and collaboration 'in the wild (McNeese, 1996).' Investigators observe users of systems in their actual work settings in an effort to understand the critical ways in which context impacts the interpretation of work and information (McNeese et al., 2000). Insights gained during ethnographic observation are then used to guide more structured cognitive fieldwork and the development of high-fidelity scenarios for use in the scaled-world simulator.

Figure 1
The Living Lab Framework (McNeese, 1996) Knowledge elicitation -or cognitive fieldwork -supports the modelling and testing of both theory and technology (MacEachren, Cai, McNeese, Sharma, & Fuhrmann, 2006). Collection of contextual data is done using established cognitive field research tools such as concept mapping and cognitive task analyses. Data collected in the field is used to inform both theory development and the development of realistic scenarios for use in scaledworld simulations where participants engage in ambiguous, incompletely understood situations of the type we find in cybersecurity (Tyworth, Giacobe, Mancuso, Dancy, & McMillan, 2012). Findings from experiments using the simulations serve as a basis for the development or modification of information and communication technology (ICT) prototypes, which can then be reintroduced into the field for evaluation.
Scaled-world simulations are a means to quantitatively assess field-based observations in a controlled setting. Because many of the other portions of the Living Lab Framework rely on qualitative methods, it can be difficult to hone in on specific constructs that may be of interest. It is very difficult to use valid field measurements to extrapolate an understanding of what is going on, without interrupting the operational workflow. Moreover, during interviews, one is forced to rely on retrospective accounts and the collection of data in a noncontrollable environment in which there are numerous potential confounds that can occur simultaneously. With this in mind, using scaled-world simulations within the living laboratory framework allows researchers to mimic the real environment and control for particular things that they are interested on. Additionally, scaled-world experiments are typically conducted in a laboratory setting, which allow for richer, quantitative, data to be collected via observations, performance measures, surveys and interviews.
Our work to date has primarily been in ethnographic observation, knowledge elicitation, and scaled-world simulation.
We have done some initial prototype development in the form of a visual analytic interface for CDA.

Ethnographic Data Collection & Knowledge Elicitation
Our ethnographic work took the form of cognitive ethnography in that it was focused on specific activities, purposive, and verifiable through triangulation of observers, data sources, and methodologies.
We conducted ethnographic observations in three contexts: a student team competing in a cyber-defence war game, professional analysts working in a large corporation, and professional analysts working in government.
We observed CDA work being done in the form of intrusion detection, forensics, and system administration. During our observation sessions we focused our attention on four areas: (1) development / lack of SA among individuals or the team; (2) collaborative activity among analysts; (3) the use of formal or informal analytical methods; and (4) cognitive breakdowns.
Typically our ethnographic observations were conducted by members of the research team working in pairs. Because of the sensitive nature of CDA work, video and audio recording of what we observed was prohibited. Instead, observers took detailed notes of their observations which were then compared by the investigators during debriefing. Where discrepancies occurred, the observer team discussed the discrepancy until agreement was reached or the discrepancy was discarded. Observations were then coded and categorized using nVivo qualitative analysis software.
The insights we gained from the ethnographic work were used to structure our knowledge elicitation activities. Knowledge elicitation consisted of semi-structured interviews of CDA subject matter experts (SMEs) in industry, government, education, and the military. We chose semi-structured interviews as our method of knowledge elicitation because the semi-structured interview format gave us enough structure to facilitate directed inquiry while allowing us the flexibility to explore emergent topics that were interesting (Spradley, 1979). Each semi-structured interview was approximately one-hour in duration and consisted of 20 to 25 predesigned questions. The predesigned questions were designed to elicit data about four areas of inquiry: routine work activity; cognitive processes associated with the development of cyber-SA; data, information, and information-processing tools used by analysts; and the influence of organizational variables such as policy, culture, and work environment on the development of cyber-SA.

Domain # of Interviews
Military 14 Government 4 Education 5 As with our ethnographic observations, the sensitive nature of CDA work prohibited us from making visual or audio recordings of the interview session. Investigators instead worked in pairs or groups of three, took detailed notes during the interview, and then debriefed after each interview. Interviews were transcribed, checked for interobserver reliability, and then coded using nVivo qualitative research software using both a priori codes and codes generated from the data. For example, our a priori codes included codes related to SA (e.g., perception, comprehension, and recognition), work (information flows, collaboration, breakdowns, tasks), and social structures (e.g., policy, norms, values). Our ethnographic and knowledge elicitation data revealed two findings that inform our on-going experimental work and prototype development. First, cyber-SA is distributed across both human operators and ICT artefacts in a complex socio-technical system spanning multiple operational domains. The domains we identified were intrusion detection, policy, operations or administration, and strategic analysis.
Though an organization's cybersecurity posture is an aggregate of all these domains, operators often have limited awareness of the environmental state of other domains.

Figure 2 Domains of CDA
For example, individuals working in the policy and operations domains are often unaware of the nature and volume of malicious activity directed towards their network because it is caught and disposed of by the network intrusion analysts without comment. This can be problematic for system administrators and in policymakers in particular, because they have incomplete understanding of the severity of the malicious activity to which their assets are exposed.
This dynamic is analogous to a battlefield commander not being aware that the enemy is probing his line because the forward operating base did not bother reporting the squad of enemy they engaged and destroyed/repulsed. In talking to intrusion detection system (IDS analysts, it became clear that the reason this dynamic occurs is because their sole focus is on identifying and blocking malicious activity. IDS analysts are not concerned about the larger threat landscape, and, in practice, the cognitive effort it takes to do that work effectively prohibits them from being able to, even if they wanted to. As one subject noted, this can be a problem because traffic may get missed for lack of a broader perspective.
The inverse of this dynamic also occurs. Individuals working in the operations and threat landscape analysis domains often do not provide mission-salient information to IDS analysts resulting in wasted effort and limited understanding. For example, it is not uncommon for an IDS analyst to diagnose traffic as suspicious only to find out upon contacting the monitored network administrators, that no, that traffic is actually approved. Similarly, a comment that repeatedly came up during interviews was that operators working in threat analysis domain would regularly ask for information on traffic from a particular device, or directed at a particular device, without providing further contextual information as to why. As a result, the IDS specialists were monitoring traffic without knowing why it was important or worthy of special attention.
The second finding is that boundaries in both physical and virtual form impair the development of system-wide SA.
The boundaries separating the functional domains are opaque; and task-salient information is only able to only partially pass through the boundaries. As a result, individuals' domain-specific cyber-SA is degraded as a result of lacking key information or knowledge from other domains. The operational domains are, in practice, heterogeneous communities of practice with their own processes, and operational languages. This finding suggests that the CDA technologies that are likely to be most effective at facilitating cyber-SA are those that function as effective boundary objects.
The boundary object is a concept that comes from the sociology work that has received widespread attention in the study of collaborative work. A boundary object is simultaneously understood by multiple communities of practice and uniquely understood by individual communities of practice (Star, 1989). Take, for example, the system log of a workstation computer. A system administrator, forensic analyst, and a policymaker all understand that the log is a file containing a record of activity on the machine, but each is capable of understanding the log in ways unique to their community of practice. A system administrator may see that the user rights were not properly secured, the forensic analyst may see the point where malware was installed on the system, and the policymaker may see where the organization is exposed to liability as a result of a data breach. Our informants repeatedly identified an interface that would function in this boundary-spanning role as something that would immediately enhance their ability to establish and maintain their understanding of what was happening in their cyber-environment.

On-going Experimental Work
We are currently conducting experiments using scaledworld simulations. One set of experiments examines transactive memory and CDA.
To conduct these experiments we have updated NeoCITIES scaled-world simulation (c.f., Jones, McNeese, Connors, Jefferson, & Hall, 2004;McNeese et al., 2005) to better support the dynamic and rich nature of the cyber security environment. The new simulation, the NeoCITIES Experimental Task Simulation (NETS), has been extended to support richer scenarios and complex decision making. The current implementation of NETS (referred to as idsNETS) has been implemented using intrusion detection data to mimic the role of an intrusion detection analyst. We have plans to extend the NETS functionality to be able to simulate scenarios from the other operational domains we identify in the future.
For our own research, we are addressing the issue of the formation and maintenance of transactive memory systems in synchronous distributed collaborations. To study this, a new version of the NETS simulation was designed (teamNETS) to simulate collaborative problem solving tasks within a cyber-environment. This version of the simulation was extended with numerous enhancements to better support our research questions and transactive memory research at large. Within the study, each team member is assigned a particular specialty, and in order to achieve high performance, it is necessary that they communicate and share relevant information to solve different types of events. From this study we hope to gain an understanding of how these transactive memory systems are formed in distributed collaborations, and how new systems can be designed to better support this process.
Transactive Memory was first conceptualized by Wegner (1985) as an "interpersonal awareness of others' knowledge" and can be conceptualized as a specialized form of Cyber Situation Awareness, where rather than focusing on, or being aware of, aspects within the cyber environment, your awareness is grounded in the cyber knowledge, activities and behaviours of your collaborators. An effective Transactive Memory System can give a human quick and coordinated access to another person's specialized expertise (Lewis, 2004). Numerous studies have shown a positive link between a team's Transactive Memory System and its performance in collaborative tasks (c.f., Ellis, 2006;Moreland & Myaskovsky, 2000;Pearsall & Ellis, 2006).
Whereas Transactive Memory is an important thread within team research it is mainly approached from a management or organization psychology lens, often only considering the humans. Since its inception, technology and information have evolved dramatically, though 7 Transactive Memory has remained fairly constant. Research has focused primarily on exploring its effect in new domains, and extending the concept as a research tool, but no one has examined how new technologies have changed how we, as humans use this transactive memory. In order to bring Transactive Memory into the 21st century, it is imperative that we understand how transactive memory has changed with synchronous distributed collaboration systems, social networks, and crowd-sourced knowledge repositories, to name a few.
A second set of experiments is being conducted to look at the impact of task load on the ability of participants to establish and maintain cyber-SA and prioritize tasks. Maintaining cyber SA is, in part, dependent on the ability to prioritize attention. Cyber defence analysts must attend to alerts associated to potential threats and respond to them within time constraints, requiring a prioritization of events in accordance to their threat level. However, high levels of cognitive workload may limit the ability of analysts to focus their attention on priority tasks. For example, unexpected surges in threat level in some events may not get noticed in time.
An interface that provides information on anticipated threat level could facilitate analysts' ability to attend to unexpected surges.
In this set of experiments we explore the effect of a workload-preview on performance in a dual-task cybersecurity event monitoring context using our NETS-DART scaled-world simulation. The simulation provides a dualtask environment. The primary and secondary tasks represent internal and external networks in an organization. All participants are presented with two types of scenarios -regular scenarios and surge scenarios. The difference between the two is that surge scenarios consist of secondary-task events that grow in threat-level and exceed that of concurrent primary-task events. Experimental results are expected to provide insight on the effect that workload previews have on attentionallocation, task management and cyber-SA in multi-task cyber-security contexts.

Prototype Development
We have developed a limited prototype of a visual analytic tool for the purpose of assessing its impact on SA. A large number of technical solutions attempt to address problems in cyber situation awareness. Typically, these solutions can be described as either data fusion technologies or visual analytic techniques. Though these two general areas of improvement will likely increase situation awareness, their impacts are seldom tested and proven in a systematic way. This part of our research seeks to answer the question of how to measure improvements in the human analyst's cyber SA. To measure improvements in cyber SA due to a data fusion or visual analytic artefact, a theoretically-grounded measurement technique must be developed specific to the cyber domain. This measurement technique must also be able to differentiate between increased cyber SA due to the knowledge and experience of the analyst from increased SA due to enhancements of the interface.

Figure 3 Visual Analytics Toolkit Prototype
As we develop the measurement technique, the simulated environment in which it is used must remain ecologically valid. The simulation that we have developed is relatively high fidelity, providing several diverse sources of cyber security data. We rely on the simulated data provided in the 2011 IEEE Visual Analytics Science and Technology (VAST) Mini-Challenge 2 including some from firewalls, intrusion detection systems, server logs and vulnerability scanners for a 3-day period over the same network (Grinstein, Whitting, Liggett, & Nebesh, 2011). These four sources of data provide a much better representation of a true cyber security environment than the singlesensor datasets previously published (Lippmann, R. P., Fried, et al., 2000;Lippmann, R., Haines, Fried, Korba, & Das, 2000;Sangster et al., 2009).
To measure SA improvement we rely heavily on SAGAT (Endsley, 1988) and its well-accepted theoretical model (Endsley, 1995b). In this project, we develop a set of freeze-probe queries to use in the simulated environment. Level 1 questions, such as which IP addresses are inside or outside of the network (D'Amico, Anita, Whitley, Tesone, O'Brien, & Roth, 2005), identify the participant's understanding of specific elements in the environment. Higher level questions probe at the memory constructs that should be present if the expected knowledge exists in the participant's working and longterm memory.
SAGAT alone, however, is unable to distinguish between SA based on knowledge and experience or whether the interface and underlying technologies provided the support for the insight. A combination of several SA measurement techniques to include the Situational Awareness Rating Technique (SART) (Taylor, 1990), National Aeronautics and Space Administration Task Load Index (NASA-TLX) (Hart & Staveland, 1988) and the Human Performance Scoring Model (Hamilton et al., 2010;Wellens & Ergener, 1988) in conjunction with a domain-specific cyber version of SAGAT should provide sufficient measurement fidelity to be able to differentiate. A 2x2 between-subjects experiment execution should provide comparison of measures between experts and novices when presented with either high or low perceived workload interfaces.
The high perceived workload interface is what would be currently available to analysts. In this interface data is correlated by IP address, but generally individual element records of the four cyber security data sources are presented in list form. In the low perceived workload interface, we present the same level of correlation, but provide the data in a visual analytic interface. This allows for the individual cyber security data records to be displayed graphically using a geographic metaphor. Host system data is placed on a "geographic map" of the network with workstations physically separated from servers and the outside Internet visually. Coordinated views in the GeoViz Toolkit (Hardisty & Robinson, 2011) provide this functionality as well as a number of powerful visual analytic representations (Giacobe & Xu, 2011).

Conclusion
To conclude, we argue for a more human-centric approach to the study of situation awareness in computer defence analysis in order to yield greater insight into the socio-cognitive challenges of CDA work.
Though valuable, much of the work done to date on situation awareness in CDA has done little to further our understanding of SA as either a cognitive state or process or empirically assessed the extent to which new technologies actually improve SA.
Our own work, which we present here, employs the Living Lab Framework to study CDA work and gain insight into both human cognitive processes related to CDA and the broader socio-technical context within which that work is done. Our findings from our field work indicate that CDA work is distributed across human actors and technological agents operating in different functional domains such as intrusion detection, forensics, and strategic analysis. We are currently engaged in multiple experiments using our scaled-world simulation -NETS -to examine questions related to transactive memory and CDA, the impact of task load on SA in CDA work, and the impact of a prototype visual analytic tool on SA in CDA work.