Nash Equilibrium Seeking with Non-doubly Stochastic Communication Weight Matrix

A distributed Nash equilibrium seeking algorithm is presented for networked games. We assume an incomplete information available to each player about the other players' actions. The players communicate over a strongly connected digraph to send/receive the estimates of the other players' actions to/from the other local players according to a gossip communication protocol. Due to asymmetric information exchange between the players, a non-doubly (row) stochastic weight matrix is defined. We show that, due to the non-doubly stochastic property, the total average of all players' estimates is not preserved for the next iteration which results in having no exact convergence. We present an almost sure convergence proof of the algorithm to a Nash equilibrium of the game. Then, we extend the algorithm for graphical games in which all players' cost functions are only dependent on the local neighboring players over an interference digraph. We design an assumption on the communication digraph such that the players are able to update all the estimates of the players who interfere with their cost functions. It is shown that the communication digraph needs to be a superset of a transitive reduction of the interference digraph. Finally, we verify the efficacy of the algorithm via a simulation on a social media behavioral case.


Introduction
The problem of finding a Nash equilibrium (NE) of a networked game has recently drawn many attentions. The players who participate in this game aim to minimize their own cost functions selfishly by making decision in response to other players' actions. Each player in the network has only access to local information of the neighbors. Due to the imperfect information available to players, they maintain an estimate of the other players' actions and communicate over a communication graph in order to exchange the estimates with local neighbors. Using this information, players update their actions and estimates.
In many algorithms in the context of NE seeking problems, it is assumed that the communications between the players are symmetric in the sense that the players who are in communication can exchange their information altogether and update their estimates at the same time. This, in general, leads to a doubly stochastic communication weight matrix which preserves the global average of the estimates over time. However, there are many real-world applications in which symmetric communication is not of interest or may be an undesired feature in applications such as sensor network. Literature review. Our work is related to the literature on Nash games and distributed NE seeking algorithms [1,17,4,11,16]. A distributed algorithm is proposed in [18] to compute a generalized NE of the game for a complete communication graph. In [7], an algorithm is provided to find an NE of aggregative games for a partial communication graph but complete interference graph. This algorithm is extended by [13] for a more general class of games in which the players' cost functions are not necessarily dependent on the aggregate of the players' actions. It is further generalized to the case with partial interference graph in [14]. For a two-network zero-sum game [5] considers a distributed algorithm for NE seeking. To find distributed algorithms for games with local-agent utility functions, a methodology is presented in [8] based on state-based potential games.
Gossip-based communication has been widely used in synchronous and asynchronous algorithms in consensus and distributed optimization problems [9,2,3]. In [9], a gossip algorithm is designed for a distributed broadcast-based optimization problem. An almost-sure convergence is provided due to the non-doubly stochasticity of the communication matrix. In [2], a broadcast gossip algorithm is studied to compute the average of the initial measurements which is proved to converge almost surely to a consensus. Contributions. We propose an asynchronous gossip-based algorithm to find an NE of a distributed game over a communication digraph. We assume that players send/receive information to/from their local out/in-neighbors over a strongly connected communication digraph. Players update their own actions and estimates based on the received information. We prove an almost sure convergence of the algorithm to the NE of the game. Unlike in the undirected case [13,14], herein we cannot exploit the doubly stochastic property for the communication weight matrix due to asymmetric information exchange. Non-doubly stochastic property leads to have total average of the players' estimates not preserved over time. This was one of the critical steps in the convergence proof in [13,14].
Moreover, we extend the algorithm for graphical games in which the players' cost functions may be interfered by any subset of players' (not necessarily all the players') actions. The locality of cost functions is specified by an interference digraph which marks the pair of players who interfere one with another. In order to have a convergent algorithm, we design an assumption on the communication digraph by which there exists a lower bound on the communication digraph which is a transitive reduction of the interference digraph. By this assumption, it is proved that all the players are able to exchange and update all the estimates of the actions interfering with their cost functions.
The paper is organized as follows. In Section II, the problem statement and assumptions are provided for the game with a complete interference digraph. An asynchronous gossip-based algorithm is proposed in Section III. In Section IV, convergence of the algorithm with diminishing step sizes is discussed. In Sections V and VI, the problem statement and the proposed algorithm for the game with a partial interference digraph are investigated, respectively and its convergence to an NE of the game is proved in Section VII. Simulation results are presented in Section VIII and concluding remarks are provided in Section IX.

Problem Statement: Game With a Complete Interference Digraph
Consider a multi-player game in a network with a set of players V . The interference of players' actions on the cost functions is represented by a complete interference digraph G(V, E), with E marking the pair of players that interfere one with another. Note that for a complete digraph every pair of distinct nodes is connected by a pair of unique edges (one in each direction).
The game is denoted by G(V, Ω i , J i ) and defined over In the following we define a few notations for players' actions.
The game is defined as a set of N simultaneous optimization problems as follows: Each problem is run by an individual player and its solution is dependent on the solution of the other problems. The objective is to find an NE of this game which is defined as follows: Note that for game (1), the NE lies in the intersection of the solutions of the optimization problems. We state a few assumptions for the existence and the uniqueness of an NE.
-Ω i is non-empty, compact and convex, The compactness of Ω implies that ∀i ∈ V and x ∈ Ω, ] i∈V be the pseudo-gradient vector of the cost functions (game map).
Assumption 2. F is strictly monotone, is Lipschitz continuous in x i , for every fixed u ∈ Ω −i and for every i ∈ V , i.e., there exists σ i > 0 such that Remark 1. Assumption 3 implies that ∇ xi J i (x) and F (x) are Lipschitz continuous in x ∈ Ω with Lipschitz constants ρ i = 2L 2 i + 2σ 2 i and ρ = i∈V ρ 2 i , respectively for every i ∈ V .
In game (1), the only information available to each player i is J i and Ω. Thus, each player maintains an estimate of the other players actions and exchanges those estimates with the local neighbors to update them. A communication digraph G C (V, E C ) is defined where E C ⊆ V × V denotes the set of communication links between the players. (i, j) ∈ E C if and only if player i sends his information to player j. Note that (i, j) ∈ E C does not necessarily imply (j, i) ∈ E C . The set of in-neighbors of player i in G C , denoted by N in The following assumption on G C is used.

Assumption 4. G C is strongly connected.
A digraph is called strongly connected if there exists a path between each ordered pair of vertices of the digraph.
Our objective is to find an algorithm for computing an NE of G(V, Ω i , J i ) using only imperfect information over the communication digraph G C (V, E C ).

Asynchronous Gossip-based Algorithm
We propose a projected gradient-based algorithm using an asynchronous gossipbased method as in [13]. The algorithm can be briefly explained as follows: -Each player maintains a temporary estimate of all players' actions.
-They receive information from the local neighbors over G C to update their temporary estimates. -Then, they solve their own optimization problems using their final estimates and update their actions.
The algorithm is inspired by [13] except that the communications are supposed to be directed in a sense that the information exchange is considered over a directed path. Our challenge here is to deal with the asymmetric communications between the players. This makes the convergence proof dependent on a nondoubly stochastic weight matrix , whose properties need to be investigated and proved. In many cases e.g. sensor networks, symmetric communication may not be of interest and could be an undesirable feature. The algorithm is elaborated as follows: 1-Initialization Step: Each player i maintains an initial temporary estimatẽ Step: At iteration k, player i k becomes active uniformly at random and selects a communication in-neighbor indexed by j k ∈ N in C (i k ) uniformly at random. Letx i (k) ∈ Ω ⊂ R N be player i's temporary estimate at iteration k. Then player j k sends his temporary estimatex j k (k) to player i k . After receiving the information, player i k constructs his final estimate of all players. Let The final estimates are computed as in the following: Note thatx i i (k) = x i (k) for all i ∈ V , since no estimate is needed for the players' own actions.
2. For all other players i = i k , the temporary estimate is maintained, i.e., We use communication weight matrix W (k) := [w ij (k)] i,j∈V to obtain a compact form of the gossip protocol. W (k) is a non-doubly (row) stochastic weight matrix defined as follows: where e i ∈ R N is a unit vector. Note that W (k) is different from the one used in [13]. The non-doubly (row) stochasticity of W (k) is translated into: Letx(k) = [x 1 (k), . . . ,x N (k)] T ∈ Ω N be an intermediary variable such that ∈ Ω N is the overall temporary estimate at k. Using (6) one can combine (4) and (5) At this moment all the players update their actions according to a projected gradient-based method.
be the intermediary variable associated to player i. Because of imperfect information available to player i, he usesx i −i (k) and updates his action as follows: if i = i k , otherwise, x i (k + 1) = x i (k). In (9), T Ωi : R → Ω i is an Euclidean projection and α k,i are diminishing step sizes such that Note that α k,i is inversely related to the number of updates ν k (i) that each player i has made until time k (i.e., α k,i = 1 ν k (i) ). In (9), the players who are not involved in communication at iteration k maintain their actions unchanged. At this moment the updated actions are available for players to update their temporary estimates for every i ∈ V as follows: At this point, the players are ready to begin a new iteration from step 2.
Now we elaborate on the non-doubly stochastic property of W (k) from two perspectives.
1. Design: By the row (non-doubly) stochastic property of W (k), the temporary estimates remain at consensus subspace once they reach there. This can be verified by (8) whenx(k) = 1 N ⊗ α for an N × 1 vector α, since, Equation (12) along with (9) and (11) imply that the consensus is maintained. On the other hand W (k) is not a column-stochastic matrix which is a critical property used in [13]. This implies that the average of temporary estimates is not equal to the average ofx. Indeed by (8), Equation (13) along with (9) and (11) imply that the average of temporary estimates is not preserved for the next iteration. Thus, it seems infeasible to obtain an exact convergence toward the average consensus [2] . Instead we show an almost sure (a.s.) convergence 1 of the temporary estimates toward an average consensus 2 . 2. Convergence Proof : λ max (W (k) T W (k)) is a key parameter in the convergence proof (as in [9,13]). Unlike [13], the non-doubly stochastic property of W (k) T W (k) ends up in having λ max (W (k) T W (k)) > 1. We resolve this issue in Lemma 2.

Convergence For Diminishing Step Sizes
In this section we prove convergence of the algorithm for diminishing step sizes as in (10). Consider a memory in which the history of the decision making is recorded. Let M k denote the sigma-field generated by the history up to time k − 1 with In the proof we use a well-known result on super martingale convergence, (Lemma 11, Chapter 2.2, [12]).
1 Almost sure convergence: Random variables converge with probability 1. 2 The same objective is followed by [9] to find a broadcast gossip algorithm (with nondoubly stochastic weight matrix) in the area of distributed optimization. However, in the proof of Lemma 2 ( [9] page 1348) which is mainly dedicated to this discussion, the doubly stochasticity of W (k) is used right after equation (22) which violates the main assumption on W (k).
As explained in the design challenge in Section 3, we consider a.s. convergence. Convergence is shown in two parts. First, we prove a.s. convergence of the temporary estimate vectorsx i , to an average consensus, proved to be the vectors' average. Then we prove a.s. convergence of players' actions toward an NE.
Letx(k) be the overall temporary estimate vector. The average of all temporary estimates at T (k) is defined as follows: As mentioned in Section 3, the major difference between the proposed algorithm and the one in [13] is in using a non-doubly stochastic weight matrix W (k) which was a key step. The following lemma is used to overcome these challenges.
) ⊗ I N and W (k) be a non-doubly (row) stochastic weight matrix defined in (6) which satisfies (7).
Due to space limitation we drop the constraints of sup(·). By the definition of Q(k), we obtain, Using (6), we expand γ as follows: Note that E[(Term 1 − Term 2) ⊗ I N ] is a symmetric matrix.
The equality only holds for x = 1 N ⊗ y where y ∈ R N and y = 1 √ N . Proof of Claim 1: Multiplying x T and x into the argument of the expected value in (17) and using x = 1, we obtain, The equality holds only when, Equation (19) holds for all k ≥ 0, i k ∈ V and j k ∈ N in C (i k ). By the strong connec- Proof of Claim 2: For x = 1 N ⊗ y and y = 1 √ N we obtain by the mixed product property of Kronecker that, It is straightforward to verify that 1 T N (Term 2)1 N = N because all the summands in Term 2 except the first one vanish by multiplying 1 T N and 1 N . Having that y T y = 1 N , (20) implies x T E Term 2 ⊗ I N x = 1 > 0. By Claims 1, 2 and using the fact that Terms 1, 2 are symmetric and γ ≥ 0, (17) implies that γ < 1.
We use Lemmas 1, 2 to showx(k) converges a.s. to Z(k). Theorem 1. Letx(k) be the stack vector with all temporary estimates of the players and Z(k) be its average as in (15). Let also α k,max = max i∈V α k,i . Then under Assumptions 1,4, the following hold.
Proof . The proof follows as in the proof of Theorem 1 in [13], but the critical step here is in using Lemma 2.
Theorem 1 yields the following corollary for x(k) andx(k).
Proof . The proof follows directly from Theorem 1 noting that x(k) = [x i i (k)] i∈V andx(k) = (W (k) ⊗ I N )x(k) (8). Theorem 2. Let x(k) and x * be the players' actions and the NE of G, respectively. Under Assumptions 4-3, the sequence {x(k)} generated by the algorithm converges to x * , almost surely.
Proof . The proof is similar to the proof of Theorem 2 in [13] based on Theorem 1.
Theorem 2 verifies that the actions of the players converge a.s. toward the NE using the fact that the actions converge to a consensus subspace (Corollary 1).

Game With a Partial Interference Digraph
We extend the game defined in Section 2 to the case with partially coupled cost functions, in the sense that the cost functions may be affected by the actions of any subset of players. The game is denoted by G(V, G I , Ω i , J i ) where G I (V, E I ) is an interference digraph with E I marking the players whose actions interfere the other players' cost functions, e.g., We also denote by N in I (i) := {j ∈ V |(j, i) ∈ E I }, the set of in-neighbors of player i in G I whose actions affect J i andÑ in I (i) := N in I (i) ∪ {i}. The following assumption is considered for G I . Assumption 5. G I is strongly connected.
The cost function of player i, J i , ∀i ∈ V , is defined over Ω i → R where Ω i = j∈Ñ in I (i) Ω j ⊂ R |Ñ in I (i)| is the action set of players interfering with the cost function of player i.
A few notations for players' actions are given: Other players' actions which interfere with J i . Given x i −i , each player i aims to minimize his own cost function selfishly, Known parameters to player i are as follows: 1) Cost function of player i, J i and 2) Action set Ω i . Note that this game setup is similar to the one in [14] except for a directed G C used for asymmetric communications.
A restatement of an NE definition (Definition 1) adapted to the interference digraph G I is defined as follows: Definition 2. Consider an N -player game G(V, G I , Ω i , J i ), each player i minimizing the cost function J i : ∈ Ω is called an NE of this game if for every given Our first objective is to design an assumption on G C such that all required information is communicated by the players after sufficiently many iterations. In other words, we ensure that player i, ∀i ∈ V receives information from all the players whose actions interfere with his cost function.
Definition 3. Transitive reduction: A digraph H is a transitive reduction of G which is obtained as follows: for all three vertices i, j, l in G such that edges (i, j), (j, l) are in G, (i, l) is removed from G.
In simple terms, a transitive reduction of a digraph is a digraph without the parallel paths between the vertices.

Remark 2.
Transitive reduction is different from the notion of maximal trianglefree spanning subgraph, which is used in Assumption 2 in [14]. Assumption 6. The following holds for the communication graph G C : Remark 3. G C is strongly connected because it is a superset of the transitive reduction of G I . Note that the transitive reduction preserves the strong connectivity of a digraph by removing only the parallel paths.
Lemma 3. Let G I and G C satisfying Assumptions 5, 6. Then,∀i ∈ V , Proof . The proof is similar to the proof of Lemma 2 in [15], but modified to adapt for the directed graph. We need to show N in For the case when G C = G I , we obtain, In (24), we used {j} ⊆Ñ in I (j) by the definition ofÑ in I (j). Now assume that G TR ⊆ G C ⊂ G I . To prove (23), it is sufficient to show that N in  (4)).
The assumptions for existence and uniqueness of an NE are Assumptions 1-3 with the cost functions adapeted to G I .
Our second objective is to find an algorithm for computing an NE of G(V, G I , Ω i , J i ) with partially coupled cost functions as described by the directed graph G I (V, E I ) using only imperfect information over G C (V, E C ).

Asynchronous Gossip-based Algorithm adapted to G I
The structure of the algorithm is similar to the one in Section 3. The steps are elaborated in the following: 1-Initialization Step: -x i (0) ∈ Ω i : Player i's initial temporary estimate.

2-Gossiping
Step: ∈ Ω j ⊂ R: Player i's temporary estimate of player j's action at iteration k. -x i j (k) ∈ Ω j ⊂ R: Player i's final estimate of player j's action at iteration k, for i ∈ V, j ∈Ñ in I (i). -Final estimate construction: For i ∈ V, j ∈Ñ in I (i), We suggest a compact form of gossip protocol by using a communication weight matrix W I (k). Let for player i, where e i ∈ R m is a unit vector. Note that W I (k) is different from the doubly stochastic one used in [14]. Using the intermediary variable, one can construct the final estimates as follows:

3-Local
Step: Player i updates his action as follows: otherwise, where T Ωi : R → Ω i is an Euclidean projection and α k,i is defined as in (10). At this moment the updated actions are available for players to update their temporary estimates for every i ∈ V, j ∈Ñ in I (i) as follows: At this point, the players are ready to begin a new iteration from step 2.

Algorithm 2
1: initializationx i (0) ∈ Ω i ∀i ∈ V 2: for k = 1, 2, . . . do 3: i k ∈ V and j k ∈ N in C (i k ) communicate. 4: W I (k) := I m − l∈(Ñ in Convergence of the algorithm adapted to G I Similar to Section 4, the convergence proof is split into two steps: 1. First, we prove almost sure convergence ofx(k) ⊂ R m to an average consensus which is shown to be the augmented average of all temporary estimate vectors. Let -m out where i : j ∈ N in I (i) is all i's such that j ∈ N in I (i). The augmented average of all temporary estimates is denoted by Z I (k) ∈ R m and defined as follows: 2. Secondly, we prove almost sure convergence of the players actions toward the NE.
The convergence proof depends on some key properties of W I and H given in Lemmas 4, 5. Proof . As suggested in (16), we employ the variational characterization of γ. Then, For the last equality, we used H T H = diag(m out ) which is straightforward to verify. We expand γ I and split the terms as follows (Let l ∈Ñ in I (i k ) ∩Ñ in I (j k )): x.
We aim to prove that x T E[Term 1]x ≤ 1. Multiplying x T and x into Term 1, we arrive at, The equality holds for all x's that satisfy H T l e s i k l (e s i k l − e s j k l ) T x = 0. After a few manipulations, by the strong connectivity of G C (Remark 3) for i ∈ V , j ∈ N in C (i) and l ∈ (Ñ in I (i k ) ∩Ñ in I (j k )) we obtain, To complete the proof we need to show x T E[Term 2]x > 0 for all x's satisfy (36) and x = 1. After some manipulations we obtain, The rest of the proof is straightforward by verifying that for all x's which satisfy (36) and x = 1, H T x = 0.
Theorem 3. Letx(k) be the stack vector with all temporary estimates of the players and Z I (k) be its average as in (32). Let also α k,max = max i∈V α k,i . Then under Assumptions 5, 6, 1 , the following hold. i) The proof is based on Lemmas 4, 5 and follows similar to the proof Theorem 1 in [15].
Proof . The proof follows by taking into account x(k) = [x i i (k)] i∈V , Z I (k) = Hz I (k),x(k) = W I (k)x(k) and also using Theorem 3.
Theorem 4. Let x(k) and x * be all players' actions and the NE of G, respectively. Under Assumptions 1 -3 , 5, 6, the sequence {x(k)} generated by the algorithm converges to x * , almost surely.
Proof . The proof uses Theorem 3 and is similar to the proof of Theorem 2 in [15].

Social Media Behavior
In this example we aim to investigate social networking media for users' behavior. In such media like Facebook, Twitter and Instagram users are allowed to follow (or be friend with) the other users and post statuses, photos and videos or also share links and events. Depending on the type of social media, the way of communication is defined. For instance, in Instagram, friendship is defined unidirectional in a sense that either side could be only a follower and/or being followed.
Recently, researchers at Microsoft have been studying the behavioral attitude of the users of Facebook as a giant and global network [10]. This study can be useful in many areas e.g. business (posting advertisements) and politics (posting for the purpose of presidential election campaign).
Generating new status usually comes with the cost for the users such that if there is no benefit in posting status, the users don't bother to generate new ones. In any social media drawing others' attention is one of the most important motivation/stimulation to post status [6]. Our objective is to find the optimal rate of posting status for each selfish user to draw more attention in his network. In the following, we make an information/attention model of a generic social media [6] and define a way of communication between users (G C ) and an interference graph between them (G I ).
Consider a social media network of N users. Each user i produces x i unit of information that the followers can see in their news feeds. The users' communication network is defined by a strongly connected digraph G C in which i → j means j is a follower of i or j receives x i in his news feed. We also assume a strongly connected interference digraph G I to show the influence of the users on the others. We assume that each user i's cost function is not only affected by the users he follows, but also by the users that his followers follow.
The cost function of user i is denoted by J i and consists of three parts: 1. C i (x i ): A cost that user i pays to produce x i unit of information.
2. f 1 i (x): A differentiable, increasing and concave utility function of user i from receiving information from his news feed with f 1 i (0) = 0.
where q ji represents follower i's interest in user j's information and L i is a user-specific parameter.
3. f 2 i (x): An incremental utility function that each user obtains from receiving attention in his network with f 2 i (x)| xi=0 = 0. Specifically, this function targets the amount of attention that each follower pays to the information of other users in his news feed. The total cost function for user i is then J i (x) = C i (x i ) − f 1 i (x) − f 2 i (x). For this example, we consider 5 users in the social media whose network of followers G C is given in Fig. 2. (a). From G C and taking J i into account, one can construct G I (Fig. 2. (b)) in a way that the interferences among users are specified. Note that this is a reverse process of the one discussed in Section 5 because G C is given as the network of followers and G I is constructed from G C . For the particular networks in Fig. 2, Assumptions 5, 6 hold. We then employ the algorithm in Section 6 to find an NE of this game for h i = 2 and L i = 1.5 for i ∈ V , and q 41 = q 45 = 1.75, q 32 = q 43 = 2 and the rest of q ij = 1. Convergence of the unit of information that each user produces to a NE over G C .

Analysis
In this section we analyze the NE x * = [0, 0, 0.42, 2.24, 0.14] T . From G C in Fig.  2 (a), one can realize that user 4 has 3 followers (users 1, 3 and 5), user 3 has 2 followers (users 2 and 5) and the rest has only 1 follower. It is straightforward to predict that users 4 and 3 could draw more attentions due to their more number of followers which is end up having less cost. This let them to produce more information (x * 4 ≥ x * 3 ≥ x * j∈{1,2,5} ). On the other hand, user 5 receives x 4 and x 3 from his news feed which ends up having greater payoff than users 1 and 2 from perceiving information. This is why x * 5 ≥ x * j∈{1,2} .

Conclusions
We proposed an asynchronous gossip-based algorithm to find an NE of a networked game over a complete graph. Then, we extended our algorithm for the case of graphical games. We specified the locality of cost functions using an interference graph. Then, we provide a convergence proof to an NE of the game under an assumption on the communication graph.