Eye-tracking Data , Complex Networks and Rough Sets : an Attempt Toward Combining Them

Eye-tracking sequences can be considered in terms of complex networks. On the basis of complex network representation of eye-tracking data, we define a measure, derived from rough set theory, for assessing the cohesion of saccade connections between object components identified in visual stimuli used in eye-tracking experiments. Rough sets are an appropriate tool to deal with rough (ambiguous, imprecise) concepts. Theoretical foundations given in the paper are supplemented with a numerical example explaining the proposed approach.


INTRODUCTION
Many important studies on eye movements in reading and information processing have appeared over the last few decades.Several general tasks, investigated by researchers, .can be distinguished, among others, reading, typing, visual search, and scene perception (cf.[14]).In case of reading, looking at a scene, or searching for an object, people continually make eye movements, called saccades.Between saccades, eyes remain relatively still during fixations for a few hundred milliseconds.Nowadays, eye movements can be automatically recorded using eye-tracking equipment (eyetrackers).Eye movement recordings deliver a dynamic trace of where a person's attention is being directed in relation to a visual display.
In the paper, we are interested in eye movements in scene perception.Viewers make many eye movements when looking at a scene.Information gleaned from detailed eye movement analyses can be used in diagnosis of various neuropsychological disorders (see, for example, [1], [8], [11]).One of the current trends is to use natural stimuli in the form of paintings instead of artificially created stimuli for eyetracking examination.This problem has been considered, among others, in one of our earlier papers [6].Perception of paintings is associated with activity of multiple regions of the brain.A structure of visual stimuli, i.e., its complexity, influences regions of the brain which are activated by visual stimuli (paintings), i.e., which cognitive functions (basic or higher) are initiated by the patient.Therefore, visual art seems to be an efficient tool in the process of diagnosis and therapy of some kinds of neuropsychological and emotional disorders.Several findings are important in detailed eye movement analyses.The first one is that the eyes are quickly drawn to informative regions of the painting.The second one is that the eyes quickly move to objects that are out of place in the scene.
In [5], we proposed to consider eye-tracking sequences obtained in the face recognition process in terms of complex networks.A proper algorithm for transformation sequences coming from eye-tracking into complex networks was given.The analysis of parameters of obtained complex networks can be helpful in better understanding and classifying human mental behaviors and activities.Transformation of eyetracking sequences into complex networks is the basis of the approach proposed in Section 3. The further analysis of eyetracking data is carried out over complex network models.
In case of scene perception, humans are able to classify the image (high-level scene classification), recognize different objects within the scene (annotation), and localize and delineate where the objects are in the scene (segmentation).These three elements are very important factors of total scene understanding (cf.[7]).On the basis of annotation and segmentation of the scene, we propose to quantitatively assess the nature of eye movements between object components identified in the scene.In the paper, we focus on cohesion of eye movements between components during the whole time of scene perception.
Rough sets proposed by Z. Pawlak [12] are an appropriate tool to deal with rough (ambiguous, imprecise) concepts in the universe of discourse.Therefore, we propose to apply rough sets to define a measure expressing the cohesion of saccade connections between object components identified in visual stimuli used in eye-tracking experiments.
The presented idea is based on the approach that uses rough sets to model some ambiguities in anticipation of states in transition and timed transition systems (which are also, in fact, graph structures), cf.[10] and [15].The measure can be defined on the basis of the original definition of rough sets [12] that is rigorous in terms of set inclusion, as well as on the basis of some extensions and generalizations of the original definition.As an example of a more relaxed and generalized rough set approach, we use the Variable Precision Rough Set Model (VPRSM), proposed by W. Ziarko in [17].The VPRSM approach was defined on the basis of the notion of the majority set inclusion instead of the standard set inclusion.

BASIC DEFINITIONS
In this section, we recall necessary definitions, notions and notation concerning both complex networks and rough sets.For more detailed information, we refer the readers to [2] (for complex networks) and [13] (for rough sets).

The Rudiments of Complex Networks
Complex networks are networks whose structure is irregular, complex and dynamically evolving in time.Formally, a complex network can be presented as a graph either undirected or directed.
In our investigations, we consider complex networks represented by undirected graphs.It means that we are not interested in directions of edges.
An undirected graph G = (N, E) consists of two sets N and E such that N = ∅ and E is a set of unordered pairs of elements of N .The elements of N = {n1, n2, . . ., nq} are the nodes of G, while the elements of E = {e1, e2, . . ., er} are the edges of G.The number of elements in N and E is denoted by q and r, respectively.The size of the graph is the number of nodes, i.e., q.In an undirected graph, each of the links is defined by a couple of nodes ni and nj, where i, j = 1, . . ., q, and it is denoted as (ni, nj).The link is said to be incident on nodes ni and nj or to join the two nodes.Two nodes joined by a link are referred to as adjacent or neighboring.
For a graph G of size q, the number of edges r is at least 0 and at most q(q−1)

2
(when all the nodes are pairwise adjacent).

The Rudiments of Rough Sets
The idea of rough sets (see [12]) consists of the approximation of a given set by a pair of sets, called the lower and the upper approximation of this set.Some sets cannot be exactly defined.If a given set X is not exactly defined, then we employ two exact sets (the lower and the upper approximation of X) that define X roughly (approximately).
Let U = ∅ be a finite set of objects we are interested in.U is called the universe.Any subset X ⊆ U of the universe is called a concept in U .Let R be any equivalence relation over U .We denote an equivalence class of any u ∈ U by [u]R.With each subset X ⊆ U and any equivalence relation R over U , we associate two subsets: Roughness of a set can be characterized numerically.To this end, the accuracy of approximation of X with respect to R is defined as: where card denotes the cardinality of the set and X = ∅.
The definitions given earlier, in some situations, seem to be too restrictive and rigorous.W. Ziarko proposed in [17] some relaxation of the original rough set approach.His proposition was called the Variable Precision Rough Set Model (VPRSM).The VPRSM approach is based on the notion of the majority set inclusion.Let U be the universe, A, B ⊆ U , and 0 ≤ β < 0.5.The majority set inclusion is defined as where card denotes the cardinality of the set.A β ⊆ B means that a specified majority of elements belonging to A belongs also to B. One can see that if β = 0, then the majority set inclusion becomes the standard set inclusion.
By replacing the standard set inclusion with the majority set inclusion in definitions of approximations, we obtain the following two subsets: called the R β -lower and R β -upper approximation of X, respectively.

ASSESSMENT PROCEDURE
In the proposed approach to assessing the cohesion of saccade connections between object components identified in visual stimuli used in eye-tracking experiments, we can distinguish three main steps: 1. Transformation of a sequence of eye-tracking data (a sequence of points in two-dimensional space) into an undirected graph representing a complex network.
2. Identification of object components in a visual stimulus and assignment of nodes of the complex network to each identified component.
3. Assessment of the cohesion of saccade connections between object components identified in a visual stimulus.
An example of a simple eye-tracking sequence over the painting is shown in Figure 1.As it was shown in [5], eye-tracking sequences can be considered in terms of complex networks.The main idea of transformation of a sequence of eye-tracking points into a complex network is to join close points, with respect to a fixed radius of circle regions of interest (ROIs), into one node in the network.The human eye cannot distinguish details located very close to each other.In Algorithm 1, for transformation of a sequence of eye-tracking points into an undirected graph representing a complex network, we use the following notation: • card denotes the cardinality of a given set, • dist is a distance function in the two-dimensional space (for example, Euclidean), • f irst is the function that gets the first element of a given set, • roi is the function that gets the index of the region of interest to which a given point belongs, • node is the function that gets the node corresponding to the region of interest with a given index.
An example of a complex network is shown in Figure 2. To display this network in the graphical form, a specialized tool called Pajek has been used (see [3]).Pajek is a program for the analysis and visualization of large networks.
Identification of object components in a visual stimulus used in the eye-tracking experiment can be performed, for example, on the basis of the approach presented in [7].Li  al. showed a hierarchical generative model that, for a given image, classifies the overall scene, recognizes and segments each object component, and annotates the image with a list of tags.If we have identified object components, we can assign, to each component, a set of nodes of the complex network which correspond to regions of interest including eye-tracking points covered by this component.
In the set of nodes of the complex network, we can distinguish: • Nodes corresponding to object components.
• Nodes corresponding to eye-tracking points not covered by any object component.Such nodes are called insignificant nodes.Let O = {o1, o2, . . ., ov} be a set of all object components identified in the visual stimulus.We use the following notation: • N = {No 1 , No 2 , . . ., No v } denotes a family of sets of nodes corresponding to object components.
• N denotes a set of insignificant nodes.
In the proposed approach, we are interested in eye movements (saccades) between identified object components.Behavior of viewers looking at a scene can be characterized by some ambiguities of eye transitions between object components.It means that fixations on some object component of the visual stimulus do not uniquely point out fixations on other object components in neighboring time moments.For example, if the eyes of the viewer were focused on the table, it would mean that they were focused earlier or later, on the vase on the table only.Such ambiguities influence the cohesion of saccade connections between components.We propose to assess ambiguities of saccade connections using rough sets.A measure based on rough sets is built over a complex network corresponding to eye-tracking data.In the presented approach, both a standard definition of rough sets and the VPRSM approach are used.
Let O = {o1, o2, . . ., ov} be a set of all object components identified in the visual stimulus used in the eye-tracking experiment and G = (N, E) be an undirected graph representing a complex network, corresponding to eye-tracking data and built by means of Algorithm 1.As it was mentioned earlier: Moreover, let us assume that we are interested in the cohesion of saccade connections between two selected object components identified in the visual stimulus, oi and oj, where i, j = 1, 2, . . ., v.More precisely, we are interested in the cohesion of saccade connections from the object component oi to the object component oj.It is worth noting that the defined measure will not be symmetrical. For we define its inter-component saccade neighborhood: Analogously to rough approximation of sets defined in rough set theory (see Section 2.2), we can define rough approximation of the inter-component saccade neighborhood (from oi to oj) over the graph G.
The lower approximation ICSN (oi → oj) of the intercomponent saccade neighborhood, from oi to oj, is given by: One can see that the lower approximation ICSN (oi → oj) of the inter-component saccade neighborhood consists of all nodes No i which are connected by inter-component edges with nodes from No j only.
The upper approximation ICSN (oi → oj) of the intercomponent saccade neighborhood, from oi to oj, is given by: The upper approximation ICSN (oi → oj) of the intercomponent saccade neighborhood consists of all nodes No i which are connected at least by one inter-component edge with nodes from No j .
The set: will be referred to as the boundary region of approximation of the inter-component saccade neighborhood from oi to oj.
If BNICSN (oi → oj) = ∅, then the approximation is exact.In the opposite case (i.e., BNICSN (oi → oj) = ∅), the approximation is rough (inexact).The accuracy of approximation of the inter-component saccade neighborhood can be defined analogously to the accuracy of approximation in rough set theory, i.e.: We treat αICSN (oi → oj) as a measure of the cohesion of saccade connections from the object component oi to the object component oj.One can see that if αICSN (oi → oj) = 1, then the connections are the most coherent ones.Let 0 ≤ β < 0.5.The β-lower approximation ICSN (oi → oj) of the inter-component saccade neighborhood, from oi to oj, is given by: The β-lower approximation of the inter-component saccade neighborhood ICSN (oi → oj) consists of all nodes No i which are connected by inter-component edges, in most cases (i.e., in terms of the majority set inclusion), with nodes from No j .
The β-upper approximation ICSN (oi → oj) of the intercomponent saccade neighborhood, from oi to oj, is given by: A relaxed measure of the cohesion of saccade connections from the object component oi to the object component oj has the form: Let us consider an example of the fragment of a complex network over the visual stimulus with identified three object components o1, o2, and o3, shown in Figure 3.We can assess the cohesion of saccade connections from the object component o1 to the object component o2 by means of the proposed approach.
In case of a standard definition of rough sets (i.e., the most rigorous case), nodes belonging to the lower approximation ICSN (o1 → o2) of the inter-component saccade neighborhood, from o1 to o2, are marked with circles in Figure 4, whereas nodes belonging to the upper approximation ICSN (o1 → o2) of the inter-component saccade neighborhood, from o1 to o2, are marked with rectangles in Figure 5. Hence, we obtain the cohesion of saccade connections, from o1 to o2, as follows: Let us calculate the cohesion of saccade connections in the opposite situation, i.e., from the object component o2 to the object component o1.Nodes belonging to the lower approximation ICSN (o2 → o1) of the inter-component saccade neighborhood, from o2 to o1, are marked with circles in Figure 6, whereas nodes belonging to the upper approximation ICSN (o2 → o1) of the inter-component saccade neighborhood, from o2 to o1, are marked with rectangles in Figure 7. Hence, we obtain the cohesion of saccade connections, from o2 to o1, as follows: One can see that a measure of the cohesion of saccade connections is not symmetrical, i.e., in general: for i, j = 1, 2, . . ., v and i = j.In case of the VPRSM approach (i.e., more relaxed case), for β = 0.25, nodes belonging to the β-lower approximation ICSN 0.25 (o1 → o2) of the inter-component saccade neighborhood, from o1 to o2, are marked with circles in Figure 8.One can see that: Hence, we obtain the cohesion of saccade connections, from o1 to o2, as follows:

CONCLUSIONS
We have shown an idea of combining eye-tracking data, complex networks, and rough sets.The combination enabled us to define a measure for assessing the cohesion of saccade connections between object components identified in visual stimuli used in eye-tracking experiments.We can distinguish two main directions of the further research.The first one will concern extending the spectrum of measures by applying various rough set approaches, among others, those based on combined rough sets and fuzzy sets (cf.[4]) or those based on probabilistic approaches to rough set theory (cf.[16]).The second one will concern deeper analysis of inter-component saccade connections with respect to semantic meaning of object components.For example, we can use the approach based on decision systems over ontological graphs (cf.[9]).
The defined measures will allow us to perform experiments assessing relationships between visual art perception and various neuropsychological disorders.One of the interesting and challenging topics is to select proper artistic styles for diagnosis of a given disorder.

Figure 1 :
Figure 1: An example of a simple eye-tracking sequence

Figure 2 :
Figure 2: An example of a complex network corresponding to eye-tracking data

Figure 3 :
Figure 3: An example of the the fragment of a complex network over the visual stimulus By replacing the standard set inclusion with the majority set inclusion in the original definitions of the lower and upper approximations of the inter-component saccade neighborhood, from oi to oj, we obtain the generalized notions of the β-lower and β-upper approximations, respectively.Let 0 ≤ β < 0.5.The β-lower approximation ICSN (oi → oj) of the inter-component saccade neighborhood, from oi to oj, is given by:

et
Algorithm 1: Algorithm for transformation of a sequence of eye-tracking points into an undirected graph representing a complex network.Input : T = t1, t2, . . ., ts -a sequence of eye-tracking points, ρ -a radius of circle regions of interest (ROIs).